86 lines
4.0 KiB
Markdown
86 lines
4.0 KiB
Markdown
---
|
|
type: blog-post
|
|
title: I am cramped
|
|
description: This post describes the ideals and values of my upcoming data platform for my homelab
|
|
draft: false
|
|
date: 2023-09-09
|
|
updates:
|
|
- time: 2023-09-09
|
|
description: first iteration
|
|
tags:
|
|
- "#blog"
|
|
---
|
|
|
|
Cramped is an experiment at building a fully open source data platform built on
|
|
both new and old components. This project is heavily biased towards my ideals,
|
|
this generally means no java (even if extremely illogical in the data world, I
|
|
will dig into this in a later post) where it can be avoided, the system should
|
|
scale, but should be able to run on somewhat resource constrained environments,
|
|
like a normal developer workstation.
|
|
|
|
My primary ideals are:
|
|
|
|
- Opinionated: I want to get 80% of the way there with 20% of the effort, if
|
|
your use-case is in the 20 remaining, then this project is not for you
|
|
- Efficient: The project should feel fast, in execution, but also to develop on.
|
|
- Minimal: The project should feel minimal, even if it has a lot of
|
|
complexities, what is done here is trying to create actual abstractions, and
|
|
let cross talk between components be as limited as possible
|
|
- Immutable and compose-able architecture: The architecture should feature
|
|
immutable steps, and present a consistent api of which real applications can
|
|
use as its bedrock. Each component should be developed in of itself, this
|
|
doesn't mean that components shouldn't communicate, but that they will have a
|
|
valid migration or use pattern in of itself.
|
|
|
|
## The name is "Cramped"
|
|
|
|
I like to use words that described a feature I'd like to avoid, I usually call
|
|
the project exactly what I am trying to do to avoid doing while developing it.
|
|
Sometimes it succeeds other times it doesn't but it is my ideal. For example in
|
|
World of Warcraft, I played a Warrior in which my play style would best be
|
|
described as _Coked out if its mind Squirrel_, aptly named Reckless, I gave it
|
|
that name to try to better my behavior, and actually play like a sane person.
|
|
|
|
Cramped is the feeling of being stuck in a small space, where even if you can
|
|
make room to do stuff, you feel constrained, annoyed and uncomfortable. I'd like
|
|
exactly the opposite, a platform where you feel like you can make the changes
|
|
you need, and iterate as you'd like, while having fun working on new and old
|
|
parts of the system, and generally trust that the changes you do makes sense.
|
|
|
|
## The goal
|
|
|
|
As mentioned Cramped is first and foremost an experiment. I've maintained an
|
|
existing data platform single-handily (to mixed success) since January (we're in
|
|
September now), as such I've not had the experience of actually building a data
|
|
platform from scratch, and with a lot of dependent engineers and analytics, I
|
|
can't really make as much experimentation as I'd like. Which gave rise to the
|
|
feeling associated with the project name.
|
|
|
|
I hope this project will help me work out some of the frustrations I've had with
|
|
our existing system, as well as get some knowledge on how to improve it, what is
|
|
current and best practice atm.
|
|
|
|
I am a platform engineer first and foremost, so definitely not a Data engineer
|
|
or Data Ops engineer. I am doing this for fun, but also go get some more
|
|
experience, so please go easy on me, as I work through and share my decisions.
|
|
|
|
## Approach
|
|
|
|
I will start by defining my top level architecture components, and simply start
|
|
small on developing each part. Each part should provide inside into which
|
|
decisions I am going for and why I am doing so. I will use esoteric languages
|
|
and tools. However, most of them have alternatives that may or may not be useful
|
|
for your platform.
|
|
|
|
Each piece will be developed by itself, and be deployed independently. I will
|
|
also use my own tools `cuddle` and `churn` for respectively
|
|
`Development Experience Platform` and `Deployment and Orchestration`. I don't
|
|
know why I have a fascination with *C*s... Each of these tools aren't ready to
|
|
see the light of day yet, but they're incredibly useful for my own development.
|
|
|
|
## Next
|
|
|
|
I will add a Table of Context of the different parts as we go.
|
|
|
|
- TBA
|