---
type: blog-post
title: "Streamlining Developer Workflows Across Your Organization"
description: "In this post, the author explores how to distribute a development platform and deployment system across an organization to share robust tools and workflows. They discuss the importance of opinionated workflows and blessed paths, which are focused on specific technologies and tooling. By choosing a blessed path, developers can share a common set of tools and reduce cognitive load, allowing them to focus on feature development."
draft: false
date: '2023-04-11'
updates:
- time: '2023-04-11'
  description: first iteration
tags:
- '#blog'
-  "Distributed Systems"
-  "Modular Architecture"
-  "Developer Workflows"
-  "Shared Tooling"
-  "Platform Teams"
-  "Scalability"
-  "Continuous Integration"
-  "Continuous Deployment"
---

In this blog post, we'll delve into distributing a continuous integration and
deployment system throughout an organization, enabling robust tools and
workflows to be shared across various languages and domains.

As an organization evolves, its developer tooling undergoes various stages based
on its size, maturity, and technology stack. A typical progression includes:

1. Creating shell scripts for project deployment.
2. Packaging shell scripts or developing basic tools, then sharing them via
   brew, copying, etc.
3. Developing dedicated tools, often by forming a dedicated operations team.
4. Establishing opinionated workflows and endorsed paths.
5. Assigning responsibilities to operations feature squads and consolidating
   them in endorsed paths.

Of course, this process isn't one-size-fits-all, and it can vary depending on an
organization's size, the products being developed, and other factors.

We'll concentrate on steps 4 and 5, examining their implications for your
company and how they can empower and accelerate your operations teams.

First, let's define opinionated workflows.

## Opinionated workflows and endorsed paths

For a medium-sized company, a smart strategy is to mature a specific tool stack
by focusing on a handful of technologies and creating exceptional tooling around
them.

It's often believed that if you're using Kubernetes, Fargate, service mesh,
etc., you can choose any language for a given service. However, many companies
have discovered that building a service involves more than just selecting a new
language; it also requires investments in infrastructure, libraries, monitoring,
expertise, and more.

For instance, if we select Java for a project where we typically build C# with
.NET, we'll find that the languages share very little in terms of tooling.

To run these services in production on Kubernetes, you'll need:

- Dockerfiles for declarative deployment
- CI tools for testing and static code analysis
- Libraries for:
  - Logging
  - Monitoring
  - Error handling
  - Circuit breaking
  - Retrying
  - API abstraction layer
  - Database ORM/querier
  - Authorization
  - And numerous other convenience libraries.

While it's not essential to build all these solutions yourself, you do need to
consider and adapt them for your needs.

Thus, it's wise to choose an endorsed path, providing developers with a
consistent set of tools and clear expectations.

An endorsed path is a predetermined service journey that automates developers'
decision-making, offering immediate and ongoing support out of the box.

You can tailor your path to your developers' needs. In our case, we designed a
journey from day-0 to day-2. For example, we didn't prioritize deprecation, but
it's acceptable for our use case, even if it involves some manual work.

This encompasses:

1. Service/Library creation
2. Dependency management, allowing teams to automatically receive updated
   dependencies
3. CI, with Jenkins jobs automatically created based on service type
4. CD, performed automatically according to service type and configuration,
   eliminating the need for teams to create Dockerfiles or Kubernetes manifests
   themselves
5. Libraries, including logging, monitoring, and more (see the library list
   above)

However, it's crucial to note that we only endorse a few languages and types and
maintain different maturity levels for them.

Think of this as developing a product centered around the platform, as I
recently discussed in a post:
[Platform Engineering: The Next Era of Ops](https://blog.kasperhermansen.com/posts/platform-engineering-the-next-era-of-ops/).

The advantages of these services include reduced cognitive load on developers
and shifting incidental complexity to expert squads. This enables feature squads
to concentrate on developing features, only tapping into the platform layer when
necessary to extend, modify, or enhance the offerings, usually with the
assistance of the platform team.

Keep in mind, this isn't a one-size-fits-all solution; it's most effective for
homogenous, large feature sprawl or services. If you have only a single product
or a few, invest time in building highly performant and tailored platforms for
them.

One issue with this architecture is the potential for unclear ownership and the
challenge of applying software development principles to platform squads. We'll
address this in the next section:

## Distributing Responsibilities for platform squads and friends

When establishing a platform organization as a product-oriented entity
interacting with the actual business organization, an API must be developed to
govern how the business domain interacts with the platform. Building this can be
incredibly challenging (think AWS or any other large product you've used).
However, unlike these organizations, you likely have direct feedback from your
users and the ability to create more opinionated solutions, allowing you to
exert more control over the workloads running on your platforms.

An alternative model involves assigning responsibilities for specific parts of
your product to each squad within your platform organization while consolidating
these responsibilities into a single API for your developers.

This results in a single tool with a wide range of capabilities, enabling:

```bash
$ platform-tool build 
$ platform-tool test
```

The challenge now lies in determining ownership for the entire build, test, and
the components that make up the pipeline. Drawing inspiration from product
development and software engineering, it becomes evident that we need to create
modules and plugin architectures for sharing these tools, allowing teams to
follow this organizational structure:

- Team responsible for the `platform-tool` itself, as well as creating
  individual commands such as build, test, etc.
- Teams responsible for building parts of the tools, such as templating for
  Kubernetes, integration testing, static analysis, code generation, etc.

There might come a time when you want to empower feature squads to develop parts
of the platform tailored to their needs, such as specific tools they've created
that necessitate a proper developer journey.

You might wonder why this hasn't been done yet or why this approach is more
challenging than it appears. The problem stems from a gap in tooling between
standard software development tools and the tools used for building the software
itself. These tools are often monolithic, require configuration, and are not
well-suited for distribution (see Dockerfiles, Helm, bash, makefiles). Most of
these products are designed to be defined within a single application, which
works for their use case but isn't scalable enough for our needs.

Large companies have gone the other way, developing highly scalable tools for
their own requirements, but these tools are generally not practical for small to
medium-sized companies to adopt (e.g., Bazel, Buck, Fabricator, etc.).
Additionally, they typically focus on a single specific use case and excel at
it.

The goal here is to introduce flexibility and autonomy into the pipeline,
enabling teams to leverage their expertise using a standard software development
paradigm.

To do so, we want to adopt a product strategy internally for the platform teams:

- The platform organization agrees on a protocol for sharing among plugins,
  templates, etc.
- The customer-facing platform tool composes these tools using the agreed-upon
  protocol.
- Each platform/feature team can own its features, which are then assembled in
  an opinionated way by the platform team responsible for the developer journey.

### Example: Golang feature service

Let's consider the example of building a Go service.

A customer-facing platform team would have already defined the main API
functions, such as build, test, code coverage, etc.

The developer journey team then composes plugins and templates into these APIs.

```go
func Build(ctx context.Context) error {
    session := ci.BootstrapSession(ctx)
    defer session.Close()

	if err := golangbin.Build(ctx, session); err != nil {
	    return err
	}

	if err := golang.Test(ctx, session); err != nil {
	    return err
	}
	
	if err := sast.Scan(ctx, session); err != nil {
	    return err
	}

	if err := docker.Publish(ctx, session); err != nil {
	    return err
	}
	
    return nil
}
```

While this is a simplified example and real-world situations would be far more
complex with numerous interdependencies, it illustrates that each team can own
their packages. For instance, a team might be responsible for the SAST plugin,
having their own submodule or repo, defining their own tests, workflow, etc.

When we publish our tool through our organization's preferred distribution
mechanism, each team can build a Go feature service using the same tooling,
automatically receiving all platform features relevant to their domain.

The same concept applies to a Node.js service:

```go
func Build(ctx context.Context) error {
    session := ci.BootstrapSession(ctx)
    defer session.Close()

	if err := nodebin.Build(ctx, session); err != nil {
	    return err
	}

	if err := node.Test(ctx, session); err != nil {
	    return err
	}
	
	if err := sast.Scan(ctx, session); err != nil {
	    return err
	}

	if err := docker.Publish(ctx, session); err != nil {
	    return err
	}
	
    return nil
}
```

Each team can use the same tooling and benefit from platform features tailored
to their specific needs, streamlining the development process and fostering a
consistent developer experience across different languages and projects. This
modular approach enables teams to focus on their core responsibilities,
delivering high-quality features and improvements while leveraging the shared
expertise and resources of the platform organization.

Despite the changes, various tools can still be reused, even if the build and
test components have evolved. While this may seem trivial to some, for those
transitioning from a shell-driven workflow, this new approach unlocks the full
potential of a traditional software development workflow, extending it to
platform squads as well.

# Conclusion

This blog post expands upon a previous post,
[Platform Engineering: The Next Era of Ops](https://blog.kasperhermansen.com/posts/platform-engineering-the-next-era-of-ops/),
by advocating for a distributed, modular approach to crafting shared tools and
workflows throughout an organization. Implementing this strategy paves the way
for a more scalable, adaptable, and efficient development process, benefiting
platform squads and feature teams alike.
