Introducing Pilum: A Recipe-Driven Deployment Orchestrator

Today I’m open-sourcing Pilum, a multi-service deployment orchestrator written in Go.

Why I Built This

After migrating SID Technologies from 15 repos to a monorepo, I had a new problem. The monorepo had Cloud Run containers, Cloud Run Jobs, Cloud Functions, and a frontend — all different service types, all needing different deployment workflows. I didn’t want 5 different infrastructure tools to deploy one repo.

At Plaid — also a monorepo — they had a ton of services and had built internal tooling around deployment that made it easy. I knew I wanted my services segmented and isolated, not a monolith, but I needed one deployment tool that understood different service types and could orchestrate them together.

So I built one.

Pilum is named after the javelin of the Roman legions — all of my products follow a Roman theme. The pilum was engineered for a single purpose: thrown once, it penetrated the target and its soft iron shank bent on impact, preventing the enemy from throwing it back. One weapon. One throw. Mission accomplished. That’s what I wanted from a deployment tool: define the target once, execute once, hit precisely.

What It Does

Pilum reads a service.yaml in each service directory, matches it to a recipe (a declarative deployment template), and orchestrates the full build-push-deploy pipeline. One command deploys any service to any supported target.

A single service:

$ pilum deploy --tag=v1.2.0

[api-gateway] ⏳ build binary
[api-gateway] ✓ build binary (2.3s)
[api-gateway] ⏳ build docker image
[api-gateway] ✓ build docker image (45.2s)
[api-gateway] ⏳ publish to registry
[api-gateway] ✓ publish to registry (12.1s)
[api-gateway] ⏳ deploy to cloud run
[api-gateway] ✓ deploy to cloud run (18.4s)

Deployment complete: 1 service deployed in 78.0s

Multiple services in parallel:

$ pilum deploy --tag=v1.2.0 --services=api-gateway,auth-service,billing-service

Step 1/4: build binary
  [api-gateway] ✓ (2.1s)
  [auth-service] ✓ (1.9s)
  [billing-service] ✓ (2.4s)

Step 2/4: build docker image
  [api-gateway] ✓ (43.2s)
  [auth-service] ✓ (41.8s)
  [billing-service] ✓ (44.1s)

Step 3/4: publish to registry
  [api-gateway] ✓ (11.2s)
  [auth-service] ✓ (10.9s)
  [billing-service] ✓ (11.8s)

Step 4/4: deploy to cloud run
  [api-gateway] ✓ (17.2s)
  [auth-service] ✓ (18.1s)
  [billing-service] ✓ (17.8s)

Deployment complete: 3 services deployed in 82.3s

All services build in parallel. All images push in parallel. All deploys happen in parallel. But each step completes before the next begins — you can’t push an image that hasn’t been built yet.

The Configuration

A service config is minimal:

name: api-gateway
provider: gcp
project: sid-production
region: us-central1

build:
  language: go
  version: "1.23"
  binary_name: api-gateway

The provider field determines which recipe is used. Recipes are declarative YAML files that define the deployment steps, required fields, timeouts, and retry behavior for each provider. You don’t write deployment scripts — you declare what you’re deploying and Pilum figures out the how.

Before any deployment runs, Pilum validates your config against the recipe:

$ pilum check

✓ Found service: api-gateway
✓ Recipe: gcp-cloud-run
✓ Required fields present: project, region
✓ Build config valid

And you can preview everything without executing:

$ pilum deploy --tag=v1.2.0 --dry-run

[api-gateway] Step 1/4: build binary
  Command: go build -ldflags "-X main.version=v1.2.0" -o dist/api-gateway .
  Working dir: services/api-gateway
  Timeout: 300s

[api-gateway] Step 2/4: build docker image
  Command: docker build -t gcr.io/sid-production/api-gateway:v1.2.0 .
  Working dir: services/api-gateway
  Timeout: 300s

...

Dry run complete. No commands executed.

How It Works

The architecture separates three concerns:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Service Config │     │     Recipe      │     │    Handlers     │
│  (service.yaml) │────▶│  (recipe.yaml)  │────▶│  (Go functions) │
└─────────────────┘     └─────────────────┘     └─────────────────┘
      WHAT                    HOW                   IMPLEMENTATION

Service configs declare what you’re deploying. Recipes define the ordered sequence of steps for a provider. Handlers implement the actual commands — building Docker images, pushing to registries, calling cloud CLIs.

Services execute in parallel within each step, with barriers between steps:

Step 1: Build Binary
┌────────────────────────────────────┐
│ service-a │ service-b │ service-c  │  ← Parallel
└────────────────────────────────────┘
════════════════════════════════════    ← Barrier
Step 2: Build Docker Image
┌────────────────────────────────────┐
│ service-a │ service-b │ service-c  │  ← Parallel
└────────────────────────────────────┘
════════════════════════════════════    ← Barrier
Step 3: Deploy
┌────────────────────────────────────┐
│ service-a │ service-b │ service-c  │  ← Parallel
└────────────────────────────────────┘

Steps can specify retry behavior with exponential backoff — useful for transient cloud errors during deployment. If a step fails for any service after retries, the entire deployment stops. No partial deployments.

For a deeper dive into the recipe system, command registry, variable substitution, and handler architecture, see the architecture docs.

Wave-Based Deployment

One of the more recent additions: wave-based deployment for dependency ordering. If Service B depends on Service A, Pilum waits for A to deploy before starting B. You define waves in your config, and Pilum respects the dependency graph while still parallelizing within each wave.

This matters for monorepos where services have real runtime dependencies — your auth service needs to be live before the API gateway that depends on it starts rolling out.

Real-World Usage at SID

At SID, we use Pilum to deploy 19 services:

$ pilum deploy --tag=v2.3.0

Step 1/4: build binary
  [authentication] ✓ (2.1s)
  [billing] ✓ (2.3s)
  [calendar] ✓ (1.9s)
  [kanban] ✓ (2.2s)
  [notifications] ✓ (2.0s)
  ... (14 more services)

Step 2/4: build docker image
  ... (19 in parallel)

Step 3/4: publish to registry
  ... (19 in parallel)

Step 4/4: deploy to cloud run
  ... (19 in parallel)

Deployment complete: 19 services deployed in 45s

Metrics after 3 months:

Pilum also deploys itself to Homebrew — the recipes/homebrew.yaml recipe handles cross-platform binary builds, archive creation, checksum generation, and pushing the updated formula to our Homebrew tap. If that recipe breaks, we can’t ship. That’s a powerful incentive to keep it working.

How Pilum Fits In

Pilum isn’t trying to replace your infrastructure provisioning or your build system. It sits in a specific niche:

Pilum’s niche: multi-service, multi-provider deployments with declarative recipes and parallel execution. If you’re deploying one service to one platform, use the platform’s native tool. If you’re orchestrating many services across multiple platforms from a monorepo, Pilum might fit.

Current Provider Support

As of this launch (v0.2.0):

• GCP Cloud Run (containers and jobs)
• Homebrew (binary distribution)
• AWS Lambda (in progress)

The recipe system is designed for extensibility — adding a new provider means writing a YAML recipe and optionally a Go handler. See the docs on creating custom recipes if you want to add support for your platform.

Known Limitations

Pilum is young. Here’s what it doesn’t do well yet:

• No built-in rollback. If a deployment causes issues, you redeploy the previous tag manually. pilum rollback is on the roadmap.
• No secret management. Pilum assumes your cloud CLI is already authenticated. Use your existing secret management solution.
• Limited observability. No dashboard, no metrics export, no Slack notifications. It’s a CLI tool that outputs to stdout.
• Recipe changes require Pilum updates. If you want to modify a built-in recipe, you need to fork or wait for a release. Local recipe overrides are on the roadmap.

These aren’t deal-breakers for our use case — small team, fast iteration. They might be for yours.

When NOT to Use Pilum

• Single service, single platform. Use the platform’s native tool. Pilum’s value is orchestration.
• Kubernetes-native workflows. Use ArgoCD or Flux for GitOps reconciliation.
• Complex build pipelines. Use Earthly or Bazel if builds are your bottleneck.
• You need rollback automation. Not yet supported.

Try It

brew tap sid-technologies/pilum
brew install pilum
pilum init          # Creates sample service.yaml
pilum check         # Validates configuration
pilum deploy --dry-run --tag=v1.0.0  # Preview

Full documentation: pilum.dev/docs

If you hit rough edges, we want to know — GitHub Issues. If you want a provider we don’t support yet, check the open provider requests or contribute a recipe.

The recipe system is designed for extensibility. We’re betting that the right abstraction — declarative recipes, pluggable handlers, parallel execution — can generalize across deployment targets. If we’re right, Pilum becomes a shared deployment layer for the ecosystem. If we’re wrong, we’ll learn something and iterate.

The code is on GitHub. The landing page is at pilum.dev.

For context on why I built this, see Managing Monorepos at Scale — the monorepo migration that created the need for Pilum.