bonofici
Menu
All posts
·6 min read

We rebuilt our operator three times. Five things survived.

The runtime that executes bonofici jobs ran on Cloudflare, then Docker, then Kubernetes, in three months. Most of it was thrown away. The parts that survived are the parts a client should demand from anyone touching their systems.

Jaume AlavedraFounder, bonofici

Every bonofici job, a month-end close, a claims decision, a shortlist, runs through an operator runtime: a sandbox that receives the standard, works inside the client's systems, and writes every step to the ledger. A named person reviews the result. We built that runtime three times in three months. v0 ran on Cloudflare, v1 on Docker, and v2 runs on Kubernetes on top of an open-source control plane. Rebuilding it that often hurt, and each failed design made the next one less naive.

This is not a post about infrastructure. It is about what turned out to be non-negotiable when you run other people's work inside other people's systems.

v0: the safe agent that could not do the job

The first version was a Worker in front of a container. Its best idea was how it handled credentials. No real key ever entered the sandbox. Every credential variable held a placeholder, a boot scan aborted the container if a secret-shaped string slipped in, and outbound requests were intercepted, checked against a deny list and then an allowlist, and only then had the real key swapped in. One audit row per decision.

It was safe, and it was nearly useless. A tight allowlist worked for a bounded job like "reconcile this account" and fell apart on anything that touched five systems. The harness was baked into the image. State lived in five different stores, so we could not rewind a session or replay it. And it cost too much to run.

v1: an orchestrator we did not mean to build

The second version used a lead agent delegating to role-specific workers, each in a fresh container with no secrets inside and a per-worker proxy injecting authentication on the wire. The recovery model was good: a dead worker was respawned and the task lived on.

The cost was time. Every task paid a 60 to 130 second cold boot before useful work began. We spent our weeks on capacity gates, heartbeats, reapers and warm pools, and none of it made the operator any better at bookkeeping.

v2: stop writing the control plane

The third version runs on an open control plane with our changes kept in a private overlay. Sessions are separate from sandboxes. A warm pool of pre-booted pods removes the cold start. Postgres holds every turn in order and enforces one active execution per thread. The sandbox receives prompt files, a command and proxy settings, and no keys.

What survived

  • Credentials never enter the sandbox. Placeholders inside, real values injected on the wire, bound to an allowed host. A compromised sandbox cannot leak a key it never held. If a provider running work in your ERP cannot explain where your keys live, that is the answer.
  • No cold starts. When every task took two minutes to begin, people stopped asking for small things. Removing the wait changed what clients handed over more than any prompt tuning did.
  • One ordered log. Every event in a session, in order, in one place. It is why we can show a client exactly what happened on a job, replay it, or redo a miss. The ledger on our site is not a marketing artefact. It is the runtime's primary record.
  • The model picks its skills. We wrote two routers and regretted both. Letting the model choose which procedure applies beat every heuristic we built.
  • Standards are plain text. Skills and standards are markdown, packed into content-addressed bundles. A client's operations lead can edit the standard and the operator picks up the new version without a deployment. Most of the infrastructure changed in three months. The plain text did not.
Hand over a function.

Get the written standard for one service before you take a call.

Book a call