Introducing Pilum: A Recipe-Driven Deployment Orchestrator

Today I’m open-sourcing Pilum, a multi-service deployment orchestrator written in Go.

Why I Built This

After migrating SID Technologies from 15 repos to a monorepo, I kept adding services. The monorepo had Cloud Run containers, Cloud Run Jobs, Cloud Functions, and a frontend. On top of that I was publishing to Cloudflare and pushing private npm packages. Every new service meant another GitHub Actions workflow pointed at another folder, copy-pasted from the last one and tweaked until it worked. Some of them worked well. Some of them worked most of the time.

Every company I’ve worked at, Plaid included, has had internal deployment tooling built for the same reason: developer efficiency. I can’t name the tools, but the goal was always one way to ship any service. As I added more services and more layers, I wanted that same consistency and reuse, and I wanted it to be fast.

I tried GoReleaser first. It handled the Go binaries fine, but it fell over when I needed to bundle a static folder into the binary so the container could serve it. So I built around it.

Pilum is named after the javelin of the Roman legions, which fits since all of my products follow a Roman theme. The pilum was engineered for a single purpose: thrown once, it penetrated the target and its soft iron shank bent on impact, preventing the enemy from throwing it back. That’s what I wanted from a deployment tool: define the target once, execute once, hit precisely.

What It Does

Pilum reads a service.yaml in each service directory, matches it to a recipe (a declarative deployment template), and orchestrates the full build-push-deploy pipeline. One command deploys any service to any supported target.

Here’s a single service deploying:

$ pilum deploy --tag=v1.2.0

[api-gateway] ⏳ build binary
[api-gateway] ✓ build binary (2.3s)
[api-gateway] ⏳ build docker image
[api-gateway] ✓ build docker image (45.2s)
[api-gateway] ⏳ publish to registry
[api-gateway] ✓ publish to registry (12.1s)
[api-gateway] ⏳ deploy to cloud run
[api-gateway] ✓ deploy to cloud run (18.4s)

Deployment complete: 1 service deployed in 78.0s

And here’s three services deploying in parallel:

$ pilum deploy --tag=v1.2.0 --services=api-gateway,auth-service,billing-service

Step 1/4: build binary
  [api-gateway] ✓ (2.1s)
  [auth-service] ✓ (1.9s)
  [billing-service] ✓ (2.4s)

Step 2/4: build docker image
  [api-gateway] ✓ (43.2s)
  [auth-service] ✓ (41.8s)
  [billing-service] ✓ (44.1s)

Step 3/4: publish to registry
  [api-gateway] ✓ (11.2s)
  [auth-service] ✓ (10.9s)
  [billing-service] ✓ (11.8s)

Step 4/4: deploy to cloud run
  [api-gateway] ✓ (17.2s)
  [auth-service] ✓ (18.1s)
  [billing-service] ✓ (17.8s)

Deployment complete: 3 services deployed in 82.3s

All services build in parallel, all images push in parallel, all deploys happen in parallel, but each step completes before the next begins. You can’t push an image that hasn’t been built yet.

The Configuration

A service config is minimal:

name: api-gateway
provider: gcp
project: sid-production
region: us-central1

build:
  language: go
  version: "1.23"
  binary_name: api-gateway

The provider field determines which recipe is used. Recipes are declarative YAML files that define the deployment steps, required fields, timeouts, and retry behavior for each provider. You declare what you’re deploying and Pilum figures out how.

Before any deployment runs, Pilum validates your config against the recipe:

$ pilum check

✓ Found service: api-gateway
✓ Recipe: gcp-cloud-run
✓ Required fields present: project, region
✓ Build config valid

And you can preview everything without executing:

$ pilum deploy --tag=v1.2.0 --dry-run

[api-gateway] Step 1/4: build binary
  Command: go build -ldflags "-X main.version=v1.2.0" -o dist/api-gateway .
  Working dir: services/api-gateway
  Timeout: 300s

[api-gateway] Step 2/4: build docker image
  Command: docker build -t gcr.io/sid-production/api-gateway:v1.2.0 .
  Working dir: services/api-gateway
  Timeout: 300s

...

Dry run complete. No commands executed.

How It Works

The architecture separates three concerns:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Service Config │     │     Recipe      │     │    Handlers     │
│  (service.yaml) │────▶│  (recipe.yaml)  │────▶│  (Go functions) │
└─────────────────┘     └─────────────────┘     └─────────────────┘
      WHAT                    HOW                   IMPLEMENTATION

A service config declares what you’re deploying. A recipe defines the ordered sequence of steps for a provider. Handlers implement the actual commands: building Docker images, pushing to registries, calling cloud CLIs.

Services execute in parallel within each step, with barriers between steps:

Step 1: Build Binary
┌────────────────────────────────────┐
│ service-a │ service-b │ service-c  │  ← Parallel
└────────────────────────────────────┘
════════════════════════════════════    ← Barrier
Step 2: Build Docker Image
┌────────────────────────────────────┐
│ service-a │ service-b │ service-c  │  ← Parallel
└────────────────────────────────────┘
════════════════════════════════════    ← Barrier
Step 3: Deploy
┌────────────────────────────────────┐
│ service-a │ service-b │ service-c  │  ← Parallel
└────────────────────────────────────┘

Steps can specify retry behavior with exponential backoff, useful for transient cloud errors during deployment. If a step fails for any service after retries, the entire deployment stops. No partial deployments.

For a deeper dive into the recipe system, command registry, variable substitution, and handler architecture, see the architecture docs.

Wave-Based Deployment

One of the more recent additions is wave-based deployment for dependency ordering. If Service B depends on Service A, Pilum waits for A to deploy before starting B. You define waves in your config, and Pilum respects the dependency graph while still parallelizing within each wave. This matters for monorepos where services have real runtime dependencies: your auth service needs to be live before the API gateway that depends on it starts rolling out.

Real-World Usage at SID

At SID, we use Pilum to deploy 19 services:

$ pilum deploy --tag=v2.3.0

Step 1/4: build binary
  [authentication] ✓ (2.1s)
  [billing] ✓ (2.3s)
  [calendar] ✓ (1.9s)
  [kanban] ✓ (2.2s)
  [notifications] ✓ (2.0s)
  ... (14 more services)

Step 2/4: build docker image
  ... (19 in parallel)

Step 3/4: publish to registry
  ... (19 in parallel)

Step 4/4: deploy to cloud run
  ... (19 in parallel)

Deployment complete: 19 services deployed in 45s

Pilum also deploys itself to Homebrew: the recipes/homebrew.yaml recipe handles cross-platform binary builds, archive creation, checksum generation, and pushing the updated formula to our Homebrew tap. If that recipe breaks, we can’t ship, so it stays working.

Where Pilum Fits

Pilum isn’t trying to replace your infrastructure provisioning or your build system. Terraform or Pulumi provisions the infrastructure, say the Cloud Run service itself, and Pilum deploys to it once that infrastructure exists, so the two work together. Tilt or Skaffold are for the development loop: local Kubernetes, hot reload. Pilum is for production, so they’re complementary rather than competing. If you’re single-language and single-platform, ko already does this well and you don’t need Pilum. If you’re all-in on Kubernetes, ArgoCD or Flux give you GitOps reconciliation that Pilum doesn’t try to do. If your builds are the bottleneck, Bazel or Earthly handle that better than Pilum does. The tool Pilum actually replaces is the pile of shell scripts that worked until they drifted.

Pilum’s niche is multi-service, multi-provider deployments with declarative recipes and parallel execution. If you’re deploying one service to one platform, use the platform’s native tool.

Current Provider Support

As of this launch (v0.2.0):

• GCP Cloud Run (containers and jobs)
• Homebrew (binary distribution)
• AWS Lambda (in progress)

The recipe system is designed for extensibility: adding a new provider means writing a YAML recipe and optionally a Go handler. See the docs on creating custom recipes if you want to add support for your platform.

Limitations

Pilum is young. There’s no built-in rollback: if a deployment causes issues, you redeploy the previous tag manually, and pilum rollback is on the roadmap but not built yet. There’s no secret management either; Pilum assumes your cloud CLI is already authenticated, so you lean on whatever secret management you already use. Observability is limited to stdout: no dashboard, no metrics export, no Slack notifications. And changing a built-in recipe currently means forking or waiting for a release, since local recipe overrides are still on the roadmap.

None of that is a deal-breaker for us: small team, fast iteration. It might matter more for you.

Try It

brew tap sid-technologies/pilum
brew install pilum
pilum init          # Creates sample service.yaml
pilum check         # Validates configuration
pilum deploy --dry-run --tag=v1.0.0  # Preview

Full documentation: pilum.dev/docs

If you hit rough edges, we want to know: GitHub Issues. If you want a provider we don’t support yet, check the open provider requests or contribute a recipe.

The recipe system is designed for extensibility. We’re betting that the right abstraction (declarative recipes, pluggable handlers, parallel execution) can generalize across deployment targets. If that holds, Pilum becomes a shared deployment layer for the whole ecosystem.

The code is on GitHub. The landing page is at pilum.dev.

For context on why I built this, see Managing Monorepos at Scale, the monorepo migration that created the need for Pilum.