206 lines
8.0 KiB
Markdown
206 lines
8.0 KiB
Markdown
---
|
|
type: blog-post
|
|
title: "Cramped: Part 1 - initial-architecture"
|
|
description: This post goes into the initial architecture and requirements of the cramped project
|
|
draft: false
|
|
date: 2023-09-10
|
|
updates:
|
|
- time: 2023-09-09
|
|
description: first iteration
|
|
tags:
|
|
- "#blog"
|
|
---
|
|
|
|
In this post I will dig into the architecture of `Cramped`. I will start by
|
|
setting the most immediate requirements for me to get started on actually
|
|
selecting components, and then finally produce a diagram of the most major
|
|
components and their communication patterns and dependencies.
|
|
|
|
In total after this post, we should have a good idea of why `cramped` exists,
|
|
which components to develop, which metrics to select for, and how the components
|
|
should be domain sliced, if they were developed by independent teams (All of
|
|
this will be developed by me however).
|
|
|
|
The overall process will be somewhat informal, and won't stick to industry
|
|
standard enterprise architect bullshit; mainly because I despise it, and have
|
|
not gotten a good result of doing it once.
|
|
|
|
## Requirements
|
|
|
|
This will be a little bit special, as I will act as both the stakeholder, and
|
|
the developer. But I am pretty hard on myself on what I want, so this should be
|
|
a sinch. I also tend to not have any respect for future _Kasper_, so this should
|
|
be fun.
|
|
|
|
1. The system should handle homogenous events, such as IoT data, application
|
|
events, and such.
|
|
2. The system should be queriable for long time analytics (3 month history),
|
|
median 95th tail latency of 30 seconds
|
|
3. The system should be used for real time analytics with a refresh window of 1
|
|
minute
|
|
4. The system should be exposeable via apis, for third party services
|
|
5. The system should be run on multiple hosts
|
|
6. Each part of the system should be deployable on its own
|
|
7. As a backend engineer, the only thing I should provide is a schema for my
|
|
event, and the upload of the data
|
|
8. As a data ops guy the only thing I is building my component, and hook it up
|
|
in a central repository for configuring where in the pipeline it goes.
|
|
9. The service should be versioned, so that schemas can evolve and old snapshots
|
|
can be compacted, to remove potential PII
|
|
10. As a data engineer, I don't want to deal with tuning the JVM, as such
|
|
alternatives should be found, this is also to not rely too much on the
|
|
knowledge I've already got doing this.
|
|
|
|
These requirements are pretty loose. we'll dig into how to actually go about
|
|
achieving these, as well as defining metrics for the ones that need it.
|
|
|
|
## Architecture
|
|
|
|
The backend services already exist, and looks like this:
|
|
|
|
### Backend
|
|
|
|
|
|
|
|
What happens here is that we have variety of business services, which are used
|
|
and serviced by either a User, or an engineer specifically. All the models are
|
|
event sourced using a home built library (something like `eventstore`, could
|
|
also be used). Some of these events are then republished as domain events, which
|
|
can be used by other services through a common communication layer, which I will
|
|
share later.
|
|
|
|
These domain events are what ends up in the data platform.
|
|
|
|
### Internet of Things
|
|
|
|
|
|
|
|
Like the backend services, we have IoT services for various things, such as
|
|
storing measured data, or controllers for doing various things. Most of these
|
|
are called or calls home assistant directly. A `Domain event` is created if
|
|
needed, which end up in the data platform.
|
|
|
|
### Existing communication layer
|
|
|
|
Services communicate with each other through both synchronous and asynchronous
|
|
messaging. For synchronous calls protobuf, json and capnp is used. json for the
|
|
frontend, usually through graphql, internally a variety of protobuf (if using
|
|
go), or capnp if using rust. Usually I don't mix go and rust services in a
|
|
single domain, as such this fits fine with my progressing into adopting capnp.
|
|
|
|
For asynchronous I've developed an in-house format based on capnp, which travels
|
|
over a database (using an outbox pattern), and then finally captured by a NATS
|
|
stream.
|
|
|
|
It sort of looks like this: written in json here to be more easily understood,
|
|
as capnp is a binary format
|
|
|
|
```json
|
|
{
|
|
"event": "user.identity.created",
|
|
"sequence": 142,
|
|
"metadata": {
|
|
"published": "<some timestamp>",
|
|
"domain": "user",
|
|
"entity": "identity",
|
|
"name": "created"
|
|
// "..."
|
|
},
|
|
"content": {
|
|
"userId": "<some-uuid>",
|
|
"email": "<some@email>",
|
|
"name": "<some full name>"
|
|
}
|
|
}
|
|
```
|
|
|
|
A schema will be supporting this data, which will also be ingested and applied
|
|
later down the line. It will also contain information on how to handle the
|
|
contents of the events, some fields may be communicated, in which the
|
|
dataplatform really, really need to be careful in handling.
|
|
|
|
```protobuf
|
|
@<id>;
|
|
|
|
using Metadata = import "cramped.metadata.capnp";
|
|
|
|
struct Event {
|
|
userId @0 :Text $Metadata.id;
|
|
email @1 :Text $Metadata.hash.sha256;
|
|
name @2 :text $Metadata.tokenize;
|
|
}
|
|
```
|
|
|
|
These annotations can evolve as well. If a change happens to a schema, the
|
|
datalake should automatically produce a new version of the data with the
|
|
changes. This may be rolling over all the data, and applying transformations
|
|
required.
|
|
|
|
These messages are sent over NATS, the event type is used as the nats routing
|
|
key, though format changed to its specific flavor.
|
|
|
|
## Data platform ingest
|
|
|
|
The data platform ingest pretty much consists of two parts. Handling events with
|
|
and without a schema / schema application failed.
|
|
|
|
An event will flow through a NATS global event consumer, which will then get a
|
|
schema applied while being transported to a data fusion pipeline (Apache spark
|
|
alternative)
|
|
|
|
|
|
|
|
This is not very different than normal data ingestion pipelines, though without
|
|
Kafka and Spark. If a schema application fails, or cannot be found, it will be
|
|
stored in the Error lake, otherwise it will be put in the data lake. through
|
|
data fusion.
|
|
|
|
Data fusion works like a sort of orchestrator, so a variety of jobs can be
|
|
scheduled on it for doing various transformations on the data lake, be
|
|
transforming the data into parquet, iceberg or something else.
|
|
|
|
It may also be used for back-filling from the error lake, or versioning /
|
|
re-partitioning the data lake.
|
|
|
|
## Data lake(house)
|
|
|
|
The idea behind the data lake is to make it a queryable place, with up to up to
|
|
date base models, and integrated transformations by the users. It includes
|
|
support for compaction, versioning, scale out queries, transformations and much
|
|
more.
|
|
|
|
|
|
|
|
In here we see the data entering the system from the NATS listener, it will pull
|
|
out data from the nats stream and ingest them into the datafusion pipeline
|
|
running in Apache Ballistra (Rust based Spark like thingy). Using deltalake as
|
|
the query and application layer, we're able to place data in the data lake in a
|
|
streamed approach.
|
|
|
|
Other jobs are run as well, for example compaction and versioning, for
|
|
automatically optimizing the query layer, as well as removing sensitive data
|
|
marked during ingestion.
|
|
|
|
A user can then query the lake directly using a regular workflow built on
|
|
deltalake for querying, the sql format is ANSI SQL, as it is queried through
|
|
Trino (distributed sql).
|
|
|
|
## Realtime analytics
|
|
|
|
I will have a need for realtime aggregations, as such a bunch of allowed
|
|
transformations should end up in clickhouse for rapid querying.
|
|
|
|
|
|
|
|
Much of the same architecture from before is used (See datalake section),
|
|
however, we also put data in `clickhouse`, this is to enable rapid querying on a
|
|
small subset and possibly sliding window of aggregations. `Trino` also
|
|
integrates with `clickhouse` for querying.
|
|
|
|
Ingest sends streamed data in much smaller batches, but still have the same
|
|
guarantees for schema application, as that is done in an earlier step. A
|
|
backfill is run regularly to fetch certain transformations, or restore precise
|
|
data consistency in the clickhouse stream. This means that technically
|
|
clickhouse can be removed entirely, and restored to a fresh instance and still
|
|
get the same data
|