This commit is contained in:
85
content/posts/2023-09-09-I am cramped.md
Normal file
85
content/posts/2023-09-09-I am cramped.md
Normal file
@@ -0,0 +1,85 @@
|
||||
---
|
||||
type: blog-post
|
||||
title: I am cramped
|
||||
description: This post describes the ideals and values of my upcoming data platform for my homelab
|
||||
draft: false
|
||||
date: 2023-09-09
|
||||
updates:
|
||||
- time: 2023-09-09
|
||||
description: first iteration
|
||||
tags:
|
||||
- "#blog"
|
||||
---
|
||||
|
||||
Cramped is an experiment at building a fully open source data platform built on
|
||||
both new and old components. This project is heavily biased towards my ideals,
|
||||
this generally means no java (even if extremely illogical in the data world, I
|
||||
will dig into this in a later post) where it can be avoided, the system should
|
||||
scale, but should be able to run on somewhat resource constrained environments,
|
||||
like a normal developer workstation.
|
||||
|
||||
My primary ideals are:
|
||||
|
||||
- Opinionated: I want to get 80% of the way there with 20% of the effort, if
|
||||
your use-case is in the 20 remaining, then this project is not for you
|
||||
- Efficient: The project should feel fast, in execution, but also to develop on.
|
||||
- Minimal: The project should feel minimal, even if it has a lot of
|
||||
complexities, what is done here is trying to create actual abstractions, and
|
||||
let cross talk between components be as limited as possible
|
||||
- Immutable and compose-able architecture: The architecture should feature
|
||||
immutable steps, and present a consistent api of which real applications can
|
||||
use as its bedrock. Each component should be developed in of itself, this
|
||||
doesn't mean that components shouldn't communicate, but that they will have a
|
||||
valid migration or use pattern in of itself.
|
||||
|
||||
## The name is "Cramped"
|
||||
|
||||
I like to use words that described a feature I'd like to avoid, I usually call
|
||||
the project exactly what I am trying to do to avoid doing while developing it.
|
||||
Sometimes it succeeds other times it doesn't but it is my ideal. For example in
|
||||
World of Warcraft, I played a Warrior in which my play style would best be
|
||||
described as _Coked out if its mind Squirrel_, aptly named Reckless, I gave it
|
||||
that name to try to better my behavior, and actually play like a sane person.
|
||||
|
||||
Cramped is the feeling of being stuck in a small space, where even if you can
|
||||
make room to do stuff, you feel constrained, annoyed and uncomfortable. I'd like
|
||||
exactly the opposite, a platform where you feel like you can make the changes
|
||||
you need, and iterate as you'd like, while having fun working on new and old
|
||||
parts of the system, and generally trust that the changes you do makes sense.
|
||||
|
||||
## The goal
|
||||
|
||||
As mentioned Cramped is first and foremost an experiment. I've maintained an
|
||||
existing data platform single-handily (to mixed success) since January (we're in
|
||||
September now), as such I've not had the experience of actually building a data
|
||||
platform from scratch, and with a lot of dependent engineers and analytics, I
|
||||
can't really make as much experimentation as I'd like. Which gave rise to the
|
||||
feeling associated with the project name.
|
||||
|
||||
I hope this project will help me work out some of the frustrations I've had with
|
||||
our existing system, as well as get some knowledge on how to improve it, what is
|
||||
current and best practice atm.
|
||||
|
||||
I am a platform engineer first and foremost, so definitely not a Data engineer
|
||||
or Data Ops engineer. I am doing this for fun, but also go get some more
|
||||
experience, so please go easy on me, as I work through and share my decisions.
|
||||
|
||||
## Approach
|
||||
|
||||
I will start by defining my top level architecture components, and simply start
|
||||
small on developing each part. Each part should provide inside into which
|
||||
decisions I am going for and why I am doing so. I will use esoteric languages
|
||||
and tools. However, most of them have alternatives that may or may not be useful
|
||||
for your platform.
|
||||
|
||||
Each piece will be developed by itself, and be deployed independently. I will
|
||||
also use my own tools `cuddle` and `churn` for respectively
|
||||
`Development Experience Platform` and `Deployment and Orchestration`. I don't
|
||||
know why I have a fascination with *C*s... Each of these tools aren't ready to
|
||||
see the light of day yet, but they're incredibly useful for my own development.
|
||||
|
||||
## Next
|
||||
|
||||
I will add a Table of Context of the different parts as we go.
|
||||
|
||||
- TBA
|
Reference in New Issue
Block a user