Kapelle A2A Architecture¶

Design doc · Draft

Teams of coding agents, each living in its own Firecracker microVM that sleeps when idle. Agents take work from Mattermost, Slack, Jira, Linear and GitHub Issues, hand work to each other over A2A, and can use any model through a self-hosted LiteLLM gateway.

Source: exported from the "Kapelle A2A Architecture" artifact. This file is the source of truth for the architecture (see CLAUDE.md).

Contents¶

1. Summary
2. Goals and non-goals
3. Constraints
4. Architecture
5. Stack
6. Transport
7. Agent runtime
8. Agent microVMs
9. Credentials
10. Models
11. Agent-to-agent communication
12. Teams
13. Surfaces
14. Team memory
15. Skills and MCP servers
16. Observability
17. Operations and security
18. Runtime evaluation
19. Decisions
20. Risks and open questions
21. Sources

Summary¶

A team is a lead agent plus roles such as coder and reviewer. It has a home room (a Mattermost channel by default, or a Slack channel) and can be linked to a Slack or Mattermost channel, a Linear team, a Jira project or a GitHub repository. Work arriving on any of these becomes an A2A task for the lead, which plans it and delegates to the other roles. Results go back to wherever the work came from.

Each agent is a long-lived Firecracker microVM running the OpenHands agent-server, with its own data disk holding repos, worktrees and conversation history. An idle agent is snapshotted to disk and its VM process stops, so it uses no CPU or memory. New work restores it, or cold-boots it from its disk. A small host daemon we write, vmd, manages the VMs. Next to it, a credential gateway adds GitHub, model and registry credentials to the agents' outgoing requests, so no credential ever enters a VM. The same stack runs on a developer's Linux laptop and on our servers, with no Kubernetes. Sandboxes sit behind a provider interface so other providers, starting with Aiven's Nevia, can be added later.

Goals and non-goals¶

Goals¶

• Create an agent team with one command, and hand it work from chat or an issue tracker.
• Agents that delegate to each other, ask questions, and deliver branches and PRs.
• Every agent runs in its own microVM, with restricted network access and no credentials inside the VM.
• Agents are long-lived and scale to zero: idle agents use disk only.
• One sandbox stack for laptops and servers, without Kubernetes.
• No lock-in to a model vendor or a sandbox provider.
• People can watch every conversation, and answer, cancel or start work.

Non-goals¶

• Publishing a standard A2A broker transport.
• Injecting messages into an agent run that is already in progress.
• Group chat between agents over A2A; broadcasts use plain NATS publish/subscribe (§11).
• Live migration of running agents between hosts.
• Forking many sandboxes from one snapshot.
• Building the Nevia integration now. The provider interface leaves room for it (§8).

Constraints¶

• No A2A in suitable agent SDKs. OpenHands and the other strong open-source coding agents don't implement A2A (§18). We add A2A around the agent runtime.
• No standard broker transport. A2A 1.0 defines only JSON-RPC, gRPC and HTTP+JSON bindings. A custom binding must cover every operation, map errors, support streaming when it declares it, and be listed in the Agent Card.
• Firecracker needs KVM. It runs on Linux x86_64 and aarch64, with read/write access to /dev/kvm. It has no virtio-fs, so workspaces are block devices, not shared folders.
• Pausing keeps memory allocated. PATCH /vm to Paused stops the vCPUs, but guest RAM presumably stays held by the VM process. Scaling to zero therefore means snapshot and stop.
• Snapshots are bound to their host. A snapshot restores only on the same Firecracker version, host kernel and CPU model (or a matching CPU template). Block devices are not part of it. Its memory file is as large as the guest's RAM. The docs call restoring the same snapshot more than once insecure.
• Restored VMs start out of step. TCP and vsock connections drop, the guest clock is stale, and metadata-service (MMDS) data must be set again.
• The agent runs inside the sandbox. OpenHands runs the agent loop and its model calls inside the agent-server, and sends whatever model API key it is configured with into it in plaintext. So the VM gets only a placeholder key, and real credentials are added outside it (§9).
• Cold boots don't resume work. After an agent-server restart, conversations that were running come back as error, and terminal (tmux) sessions are gone. Nothing restarts automatically.
• OpenHands has no "needs input" status. A clarifying question ends a run as finished (§7).
• Events stay on one replica. a2a-sdk ships only an in-memory event queue manager.
• Surface limits.
• Slack: chat.postMessage about 1 per second per channel.
• Mattermost: posts are limited to MaxPostSize characters (read from GET /api/v4/config/client?format=old); the API rate limit is off by default and, when on, 10 requests per second with a burst of 100 per client address.
• Linear: a thought within 10 seconds of a session starting.
• Jira: dynamic webhooks expire after 30 days.
• GitHub: issues can't be assigned to an App.

Architecture¶

Boxes marked custom are code we write. The dashed teal outline is the sandbox boundary.

flowchart LR
    subgraph Surfaces["Surfaces"]
        Mattermost["Mattermost"]
        Slack["Slack"]
        Linear["Linear"]
        Jira["Jira"]
        GH["GitHub Issues"]
    end

    SG["Surface gateway<br/>(custom — one adapter per surface)"]
    Mattermost <--> SG
    Slack <--> SG
    Linear <--> SG
    Jira <--> SG
    GH <--> SG

    TC["Team controller (custom)"]
    PG[("Postgres<br/>teams · agents · tasks")]
    SG -->|"/team"| TC
    TC -->|"teams · agents"| PG

    NATS[["NATS JetStream<br/>KV a2a-cards<br/>team.&lt;t&gt;.a2a.&lt;role&gt;.rpc<br/>….&lt;role&gt;.cancel<br/>team.&lt;t&gt;.tasks.&lt;id&gt;.…<br/>team.&lt;t&gt;.activity.…"]]
    SG <-->|"A2A"| NATS
    TC -->|"cards"| NATS

    subgraph RW["Role worker (stateless, outside VMs)"]
        A2AS["A2A server (custom)<br/>NATS adapter · a2a-sdk"]
        TOOLS["A2A tools (MCP) (custom)"]
        EXEC["AgentExecutor · sandbox provider (custom)"]
        A2AS --> EXEC
    end
    NATS <-->|"A2A"| A2AS

    subgraph HOST["VM host (Linux + KVM)"]
        VMD["vmd (custom)<br/>Firecracker API"]
        CG["Credential gateway (custom)<br/>adds credentials"]
        VMD --> CG

        subgraph VM["Agent microVM (sandbox boundary)"]
            AS["agent-server<br/>OpenHands"]
            DISK[("data disk<br/>repos · worktrees · state")]
        end
    end

    EXEC -->|"wake · sleep"| VMD
    EXEC -->|"REST · WebSocket"| AS
    TOOLS -->|"MCP via gateway"| CG

    LLM["LiteLLM proxy<br/>keys · budgets · aliases"]
    VLLM["vLLM<br/>+ commercial APIs"]
    BAO[("OpenBao")]
    OTEL["OTEL collector"]

    CG -->|"+ agent key"| LLM
    LLM --> VLLM
    BAO -->|"short-lived tokens"| CG
    CG -.-> OTEL

Workers hold no agent state. For each task they look up the agent handling the work item, ask the sandbox provider to wake its microVM, then drive its agent-server. The VM never connects to NATS or Postgres, and holds no credentials. Everything it sends out goes through the credential gateway on its host, which adds LiteLLM, MCP, GitHub and registry credentials from OpenBao (§9).

Not drawn:

• the git host and package registries, reached through the credential gateway
• the OpenTelemetry export from agent-servers and workers to the collector
• the team memory service, which lives in the controller and Postgres and is reached through the worker's MCP tools (§14)
• workers' own writes to Postgres
• Jira's remote-agent calls, which reach the lead's standard A2A HTTP interface through the gateway

Stack¶

Versions checked on 2026-10-06 (versions.yaml is the pin list, just versions the check); the rule is to run the current release of every component (images, actions, modules) and to use Valkey wherever a Redis-compatible store is needed. Control-plane services (gateway, controller, workers, NATS, Postgres, LiteLLM, Mattermost) are ordinary containers, run with Compose on a laptop or as containers or systemd services on servers. Only vmd and Firecracker run directly on KVM hosts.

Transport¶

Messages on the bus are A2A 1.0 JSON-RPC envelopes using the spec's method names (SendMessage, GetTask, CancelTask, …); only delivery is custom. Every subject is scoped to a team. Only workers, the gateway and the controller connect to NATS; agent VMs never do.

Rules¶

• Server adapter. Reads the rpc subjects, calls RequestHandler methods and publishes the resulting events to the task's events subject.
• Client transport. Registered with ClientFactory.register(label, producer). Agent Cards list the NATS interface and the HTTP interface.
• Ack at SUBMITTED. Acknowledge a JetStream message once its task is saved, not when the task finishes. Waking an agent can take longer than the ack deadline.
• Task leases. Once acked, the task belongs to the worker that holds its lease in Postgres, renewed by heartbeat while the run is in progress. A reconciler re-dispatches tasks whose lease has expired (a crashed worker), or fails them after a retry. Without this, a task stays WORKING forever. A worker whose heartbeat finds the lease gone (taken over by another worker, or its row cleared) stops the run and lets go of the task: it writes no terminal state, publishes no event or reply, opens no pull request and hands nothing on, and the worktree and conversation stay for a re-dispatched run to reattach to. A push that had already happened before the loss was noticed stays (the branch is keyed by task id and the retry pushes it again, and its pull request is reused). The worst case is a branch tip from a run that never reported.
• Replies are messages, not subscriptions. When a delegated task reaches INPUT_REQUIRED or a terminal state, the worker driving it also publishes a SendMessage to the requesting role's rpc subject, with the requester's contextId and the delegated task id in metadata. The reply is then a durable, queued message that any worker can handle, rather than something held in one worker's memory (§11).
• Cancels go to every worker. Event queues are in-memory, so only the worker driving a task can cancel it. If none holds the lease, the reconciler marks the task CANCELED directly.
• Shared task store. DatabaseTaskStore. Run a2a-db migrations on every SDK upgrade.
• Backpressure. max_ack_pending matches the number of concurrent runs the team's agents allow (§12).
• Retention. The events and activity streams have size and age limits (for example 30 days), and Postgres holds the durable task record. Otherwise JetStream grows without bound.
• Permissions. Workers are shared across teams, so team boundaries are enforced by the agent-scoped MCP token (§7). Teams with isolation: dedicated also get their own NATS account (operator → account → user, decentralized JWT auth): subject space and JetStream streams/consumers are independently namespaced per account, and each dedicated team's user JWT further scopes pub/sub to that team's own subjects plus its reply inboxes. Provisioned at TeamService.create() time, credentials delivered via OpenBao like every other per-team secret; see contracts/a2a-nats.md §"Dedicated NATS accounts" for the exact permission set and minting flow, and docs/team-isolation.md for what's implemented today versus not yet (backlog task d74d9c67).

Agent runtime¶

For each task, AgentExecutor in a stateless worker does the following:

1. Looks up which agent replica handles the contextId. A work item stays with the replica that holds its worktree and conversation.
2. Calls provider.wake(), which restores or boots that agent's microVM and returns an agent-server endpoint.
3. Creates or reattaches the work item's conversation.

The agent loop runs inside the VM; the executor follows its events.

endpoint = await provider.wake(agent_ref)  # restore snapshot or cold boot
llm = LLM(
    model=f"litellm_proxy/{role.model_alias}",  # alias defined in LiteLLM
    base_url=GATEWAY_LLM_URL,  # the host's credential gateway
    api_key="placeholder",  # real key added by the gateway
)
agent = Agent(
    llm=llm,
    tools=[Tool(name=TerminalTool.name), Tool(name=FileEditorTool.name)],
    # + the worker's A2A tools as an MCP server at
    #   {GATEWAY_MCP_URL}/{conversation_id}   (config field: not verified)
)
workspace = RemoteWorkspace(host=endpoint.url, api_key=endpoint.session_key)
conversation = Conversation(
    agent=agent,
    workspace=workspace,
    conversation_id=uuid5(TASK_NS, f"{context_id}/{role_name}"),
    delete_on_close=False,  # default True deletes server-side state
    max_iteration_per_run=role.max_iterations,
    callbacks=[publish_activity],
)
conversation.set_confirmation_policy(NeverConfirm())  # the microVM is the security boundary
conversation.send_message(task_text)
conversation.run()

One agent-server hosts every conversation its agent holds. Each work item gets its own git worktree: the executor creates the conversation with POST /api/conversations and worktree=true, since RemoteConversation doesn't send that field. If the path isn't a git repo, the server silently falls back to the shared checkout, so the executor checks the returned workspace.

How OpenHands status maps to A2A¶

Rules¶

• Delegation and memory tools. list_agents, send_task, get_task, ask_requester and the memory tools (memory_search, memory_get, memory_propose, §14) run on the worker as an MCP server. Each conversation is configured with its own MCP URL that carries the conversation id, since one VM can run several work items at once and the gateway can't tell them apart by network source. The credential gateway adds a token scoped to the agent; the worker maps the conversation id to the task. The VM holds no token and no NATS credentials. send_task returns without waiting.
• Push after every run. The role prompt and a post-run hook push the work item's branch to the git host whenever a run ends, so a lost data disk costs caches and conversation history, not code.
• Branch handoff between roles. send_task attaches the delegator's own CURRENT working branch to the outgoing A2A metadata — whatever was handed to the delegator, or a freshly minted agent/<team>/<delegatorTaskId> if its role produces one of its own (working_branch_for, backlog task df449689) — attached only when the TARGET role's own roles/<role>.md front matter says it works_on_branch. This durable value is persisted once, on the receiving task's own task_origin row at creation, and every later reader (that task's own checkout, and any further delegation IT makes) reads it from there — never from a later turn's own message metadata, since a resumed/follow-up turn of the same task can arrive with no metadata at all. This is what keeps a rework round on the SAME branch and pull request as the round it's reworking, rather than minting a second one keyed off the rework task's own (different) task id — the original shape of this design named the branch agent/<team>/<callerTaskId> unconditionally and recomputed it independently at each push/PR/delivery site, which is exactly what produced that bug. The receiving role fetches that branch before its first turn, retrying for a bounded window: the delegator's own push happens at the end of its run, strictly after send_task already returned, so the branch may not exist yet the instant the receiving role's conversation starts. It never checks the branch out by name — every conversation's worktree is created once by the OpenHands SDK on its own private branch (openhands/<conversationId>, unique per conversation) and stays there for its whole life; nothing needs the shared name locally (a reviewer's own diff is git diff $(git merge-base HEAD origin/main), and a push names its source by commit, not by branch). Instead the fetched commit is brought in by position: already contained in HEAD, nothing to do; HEAD behind it, fast-forward; HEAD holding nothing of its own (every commit already on the remote, checked with no assumed branch name), move the worktree's own branch onto it; otherwise a genuine divergence, failing and naming both commits (push.fetch_branch_into_worktree). This design replaced an earlier one that checked the branch out by its real name — safe on its own, but two different roles' worktrees handed the identical branch (a rework round pushing back to the same coder branch a reviewer still held) collided outright, since git never lets two worktrees of one repository hold the same branch checked out at once; never creating the shared name locally removes the collision instead of tolerating it. send_task itself is idempotent per (callerTaskId, role): a second delegation call for the same pair returns the already-delegated task id rather than minting a new one, regardless of what the retried call's message text says. A reply carrying kapelle.reply_to_task is a resumption of the requester's own conversation, never a new delegation origin in its own right — its own eventual completion must not trigger a further reply back. A COMPLETED reply is only published when the requester's own task is genuinely INPUT_REQUIRED — paused, idle, waiting to be resumed by exactly this reply (backlog task 3e9386a1: a merely non-terminal SUBMITTED/WORKING requester is not "waiting" on this reply, it is actively mid-run on something else, and since only one run at a time is allowed per conversation the reply can't join that run anyway — it would only mint a wasted new task). Once a role has handed off and finished, or is simply still busy with its own remaining work, a completion notification would only start an empty (or duplicate) turn, so it is dropped. FAILED, CANCELED, REJECTED and INPUT_REQUIRED replies always go through, because someone has to react to them.
• Non-blocking drive. Workers start runs and follow them over the WebSocket event stream instead of holding a blocking run() thread per task. The stream reconnects automatically (backoff capped at 30 s), and reconnect and read timeouts are longer than the slowest expected wake.
• Subagents vs A2A. OpenHands delegate and task tools split work inside one agent. A2A is for work between roles.
• max_iterations is a per-turn budget, not a lifetime one. role.max_iterations is passed straight through as max_iteration_per_run (the SDK's own name for it), and the SDK's own run loop resets its iteration counter to 0 at the start of every run()/arun() call — a work item resumed many times over its life (every delegate reply reattaches the same conversation) gets the role's full budget again on each turn, not a shrinking share of one lifetime total. run() raises instead of returning normally when a run ends error/stuck (including an ordinary MaxIterationsReached) — the executor must catch that and classify it the same way as any other terminal status, or the run crashes uncaught instead of failing cleanly, which from the outside looks exactly like a lifetime cap that never resets (a real regression found and fixed, backlog task 77e6819b). Each role's own max_iterations still needs to fit what a SINGLE turn of that role actually does — a planner that reads a repo with read/grep/glob before delegating spends most of a small budget just on exploration, so tune it around one turn's worst realistic case, not around "the whole work item."
• Pin versions. OpenHands shipped 1.45 to 1.47 between 7 and 10 September. The SDK and the VM image are pinned together and upgraded behind the evaluation suite (§18).

Agent microVMs¶

Each agent, for example payments/coder-1, is one long-lived Firecracker microVM. Its data disk holds everything the agent keeps between work items. When idle, the agent sleeps: a full snapshot goes to host storage and its Firecracker process stops. On new work, vmd restores it. If there is no usable snapshot, vmd cold-boots it from its disks. The same vmd runs on a developer laptop and on servers.

What is in an agent VM¶

Lifecycle¶

stateDiagram-v2
    [*] --> Provisioned
    Running: Running (RAM + CPU)
    Idle: Idle (RAM, no runs)
    Asleep: Asleep (disk: memory file + data)
    DeepSleep: Deep sleep (disk: data only)

    Provisioned --> Running : cold boot
    Running --> Idle : runs = 0
    Idle --> Running : work
    Idle --> Asleep : idle timeout
    Asleep --> Running : work arrives — restore snapshot (single use)
    Asleep --> DeepSleep : deep-sleep timeout, upgrade, host move
    DeepSleep --> Running : work arrives — cold boot from disks

Teal states (Asleep, Deep sleep) use disk only. An agent goes to sleep only when no conversation is running, so a cold boot loses terminal sessions but no work in progress.

Sleep¶

1. GET /api/conversations/count?status=running returns 0, as does the waiting_for_confirmation count for roles that use confirmations, and /server_info reports idle_time above the idle timeout.
2. Mark the agent as sleeping in Postgres, so new work waits for the wake instead of racing the snapshot.
3. Let the balloon return freed memory. Whether this shrinks the full memory file is not yet measured (§20).
4. PATCH /vm {"state": "Paused"}, then PUT /snapshot/create with snapshot_type: Full to encrypted host storage. Flush the data disk.
5. Stop the Firecracker process and release the tap slot.

Wake¶

1. Start a new Firecracker process through the jailer, with the same drive paths. Use network_overrides if the tap name changed.
2. PUT /snapshot/load with mem_backend: File (loads memory on demand) and resume_vm: true. On failure, fall back to a cold boot.
3. Delete the snapshot files once the resume succeeds; snapshots are never restored twice.
4. Fix the clock before the guest can send anything: clock_realtime on /snapshot/load (x86_64) is applied before resume; on aarch64 the KVM PTP clock plus chrony in the guest must sync first, so the tap stays down until a guest agent reports the clock as synced. A stale clock fails TLS certificate and token expiry checks at the gateway.
5. Write MMDS data again, bring the network up, and wait for the agent-server's /ready. The executor reattaches by conversation_id.

Known limitation: step 5's MMDS rewrite has no effect on a restored guest today — kapelle-guest-agent only ever reads MMDS once, at cold-boot time, and doesn't re-read it on restore (there's no reconfigure signal wired up; see vmd/README.md). This doesn't matter in practice: Ensure refuses to change an already-booted agent's session key, and a gateway's address is fixed for the agent's whole lifetime (deterministic per its own /30), so nothing MMDS carries actually changes across a restore. If that ever stops being true, a restore will silently keep serving the old values until the agent is cold-booted — building a real reconfigure-in-place mechanism (a vsock ping triggering a guest-side MMDS re-read and openhands-agent-server restart) is deferred until something needs it.

Cold boot (deep sleep, a failed restore, a Firecracker or host kernel upgrade, or moving hosts): boot from the root image and data disk. Conversations reload from disk. Snapshots are not carried across upgrades.

Measured performance¶

From the Firecracker spike (backlog spike 2, a minimal stdlib guest, 2 vCPU/512 MiB, this dev hardware — see spikes/firecracker/FINDINGS.md), medians over several runs, network-reachable-and-agent-ready:

Restore is ~3.3x faster than cold boot on this hardware even before accounting for the real agent-server's own startup cost below — most of a cold boot's time is the guest's own network bring-up and process startup, both of which a restore skips entirely (network state and process memory both survive in the snapshot).

Against the real agent image (backlog task 10, openhands-agent-server 1.47.0, full boot-to-/ready including systemd, chrony, and the Python/SDK import): ≈ 25.3 s → ≈ 9.2 s after fixing chrony's PHC reference-clock configuration. The guest's kvm-clock/PHC device delivers TAI, not UTC (currently ~37 s ahead — the accumulated leap-second count), and the original chrony config polled every 8 s with no offset compensation, so chrony spent most of the boot slowly discovering that gap through repeated samples instead of correcting for it up front. Switching to poll 0/dpoll -2/offset -37 (chrony's own documented recipe for this exact device) cut source selection from ~24.8 s to ~3.8 s. The new dominant cost is openhands-agent-server's own Python/SDK import time, ~5.3 s of the ~9.2 s total — not yet optimized, the next thing worth profiling (see images/README.md).

Data-disk staging (decrypting/encrypting directly into the jailer chroot instead of through an intermediate host-side scratch copy, see What is in an agent VM above): for a 1 GiB disk, 30-55% faster depending on concurrent host load, and faster in every sample measured — eliminates one full-size copy of the disk's contents per boot, restore, and sleep.

Measured on thor, 2026-09-13 (backlog task 34, vmd/bench_real_test.go's TestMeasureWakeAndMemoryRealImage, N=5 real cold-boot/sleep/restore cycles, real agent image agent-python-bbdfb30bfb98.ext4, real production kernel, 2 vCPU/2048 MiB, 512 MiB data disk): "ready" is the guest's own agent_server_ready=true, not just vmd's Wake() returning; "memory" is the real Firecracker process's own RSS from /proc, read at ready and again after 60 s of guest idle.

Cold boot cross-validates closely against the independent ~9.2 s measurement above (different harness: images/test-boot.sh drives Firecracker directly, this one goes through vmd's own Ensure/Wake). Restore's RSS is genuinely far lower than cold boot's, not a measurement artifact: a restored process's memory-mapped mem_file is demand-paged, so RSS only reflects guest pages actually touched since resume, and climbs further as background activity (chrony, systemd, the agent-server itself) touches more of it during the idle window — a host doing mostly restores should expect meaningfully lower resident memory per agent than a naive mem_size_mib estimate assumes. The plaintext mem file Firecracker writes is pinned to guest RAM regardless of actual usage (Full snapshots are always dense at mem_size_mib, spike 2 finding) — but vmd now zstd-compresses it before encrypting (backlog task 37, internal/snapcompress), since most of that density is genuinely untouched guest RAM: the same real N=5 run above compressed to a median 142.0 MiB on-disk bundle (≈14.4x), with no measurable Sleep/Restore latency cost. Diff snapshots (Firecracker's other size lever) were investigated and not implemented — they'd need to retain a base snapshot indefinitely, contradicting today's single-use semantics, for a smaller incremental win on top of compression. Full raw samples and methodology: docs/eval-results.md.

Disk lifecycle¶

• Cleanup. A long-lived agent would otherwise keep a worktree, branch and conversation directory for every work item forever. The controller computes a cutoff (7 days after a task reaches a terminal state) and calls vmd's Cleanup RPC, which forwards it to kapelle-guest-agent's own POST /cleanup — vmd never mounts or inspects the data disk itself, so "the VM is the only thing touching its own disk" holds even for cleanup. kapelle-guest-agent deletes each qualifying conversation through the real agent-server API and removes and prunes its worktree; the branch stays on the git host.
• Cap. Each data disk has a size cap from its SandboxSpec. When usage passes 80%, vmd runs cleanup early and posts a notice in the team room; at the cap, the agent stops accepting new work items until space is freed.
• Backups. Data disks are copied to object storage on deep sleep, before archive, and on a schedule (nightly) for every sleeping agent, so a host loss costs at most a day of caches and conversation history. Code is safe earlier because branches are pushed after every run (§7).
• Encryption. Data disks are encrypted at rest with a per-agent key held in OpenBao, like snapshots — a chunked AES-256-GCM container (vmd/internal/crypt, the same format snapshots use), not dm-crypt/LUKS: equivalent confidentiality without a loop-device/kernel-dm-crypt dependency inside the jailer chroot, and the disk is plaintext only while staged into the running VM's chroot. Measured cost on the dev host: encrypting or decrypting a 1 GiB disk takes roughly 400-800ms depending on concurrent host load (AES-NI bound, ~2.5+ GiB/s on otherwise-idle hardware) — negligible next to Firecracker boot/snapshot time, and linear in disk size. They contain source code and conversation history.
• Disk cap enforcement, disk usage, and cleanup are split across two components. vmd polls the guest's disk usage and reports it on Status/Capacity; it does not decide to warn or stop accepting work itself — that decision, and the actual cleanup, are the controller's and the guest's respectively, per "the VM is the only thing touching its own disk" below.

vmd, the host daemon¶

• Responsibilities:
• root image cache and per-agent data disks
• starting Firecracker through the jailer
• network slots (tap in the host's root network namespace, per-tap nftables rules — see below)
• MMDS data
• snapshot, restore and cold boot
• idle detection and sleep
• a watchdog that kills hung VMM processes
• orphan cleanup on startup: a jail or snapshot directory with no matching local-state record (left by a previous vmd instance that crashed or was killed before it could Destroy them) gets its process killed, its files removed, and its tap/nftables rules torn down
• refuses to start if its jail or snapshot directory resolves to a tmpfs (RAM-backed) filesystem — staging a large data disk or snapshotting a large VM's guest memory there can exhaust host RAM outright
• warns at startup if the host's own firewall (e.g. ufw) manages a separate nftables table with a DROP policy on the input hook and no rule covering the configured network pool — best-effort, read-only, never modifies the firewall (see "Egress" below and docs/dev-environment.md's troubleshooting section)
• metrics
• API. gRPC over mTLS to workers and the controller: Ensure, Wake, Sleep, Destroy, Status, List, Capacity, Cleanup, Touch (extends how long the idle detector must wait before sleeping a sandbox, for a raw operation like a pre-conversation repo clone that has no "conversation" of its own to be idle in), NetDiag (a read-only, plain-text dump of one agent's network path — tap counters, per-tap nftables rule counters, gateway listener state, allocated addressing, touch/idle status — for vmdctl net-diag, see docs/dev-environment.md's troubleshooting section).
• Language. Go, so we can borrow from E2B Runtime's orchestrator (Apache-2.0): per-sandbox network namespaces with nftables and an SNI egress firewall, userfaultfd resume, copy-on-write root disks. We skip its API layer, ClickHouse and Nomad. In the end we dropped the per-sandbox network namespace ourselves (below) — we never clone VMs, so netns-per-VM isolation bought nothing a per-tap nftables rule didn't already give us, at the cost of a veth pair and an extra table per VM.
• Placement. The controller places an agent on a host with enough RAM for running agents and enough disk for sleeping ones. A sleeping agent is tied to the host holding its snapshot; deep-sleeping agents can move by copying the data disk: vmdctl migrate -agent-id <team>/<role>-<replica> -to <host> (backlog task f194960c) backs up the DEEP_SLEEP agent's data disk and destroys its local state on the source vmd (the existing Destroy(SkipBackup=false) RPC, no protocol change needed), then calls the controller's ReassignAgentHost RPC (contracts/proto/controller.proto) to point that agent_id at the target host — its next Ensure/Wake lands there and transparently restores the backup.
• Laptop. vmd runs natively (it needs /dev/kvm and permission to create taps and nftables rules) — as a plain, non-root process: every privileged operation execs one pinned binary (ip, nft, jailer, firecracker) through a helper that transparently prefixes sudo -n when not already root. deploy/host/sudoers.kapelle (generated by vmdctl sudoers from internal/host.PrivilegedCommands, the exhaustive list of every command+argument shape vmd's own code issues) grants exactly those shapes — not the bare binaries, which would be effectively unrestricted root for anything running as that user. The Cmnd_Alias it defines is named by vmdctl sudoers -alias (default KAPELLE_VMD) — sudo merges every file under sudoers.d into one namespace, so this name must differ from any other sudoers file already installed on the host, or visudo/sudo rejects the install as a duplicate alias (found for real colliding with an operator's own pre-existing /etc/sudoers.d/kapelle-dev). See docs/dev-environment.md's "Running vmd/credgwd natively" section for the actual install commands. Control-plane services run in Compose. A real production host follows the identical shape, not root: vmd.service runs vmd as a dedicated, unprivileged vmd system user (backlog task 0ca6d44f), so the privileged-exec helper takes the same sudo -n branch there too — deploy/host/provision.sh installs a separate, production-shaped grant (sudoers.kapelle.prod, vmd's real paths, no test roots) as a live dependency, not defense in depth; see deploy/host/README.md's "Sudoers" section. This is now proven on a real (nested-KVM) host, not just asserted: just prod-vm-up/prod-vm-verify/prod-vm-down (backlog task 0ca6d44f, 8d2d809) run provision.sh --apply for real inside a fresh Ubuntu 24.04 guest and drive a real Ensure/Wake/Sleep/Wake/Destroy round trip against vmd running as that non-root user — the run itself found and fixed three real sudoers gaps this dev workstation's own Arch/usrmerge layout never exercised. One real compatibility boundary this surfaced: Ubuntu 26.04 LTS's new default sudo-rs implementation hard-rejects the wildcard-shaped grants this whole scheme depends on, so a provisioned host needs traditional sudo until a privilege helper that doesn't need wildcard sudoers grants lands — not something fixed here, and why the verification recipe deliberately targets 24.04 LTS instead. That helper has since landed (backlog task a8b405aa): vmd-priv, a small root-owned server (vmd/cmd/vmd-priv) exposing exactly PrivilegedCommands over its own Unix socket, argument-validated in Go (host.ValidatePrivilegedCommand, the identical resolvePattern/matchGlob RenderSudoers itself uses, so the two schemes can never authorize a different shape for the same table) and peer-authenticated the same internal/unixcreds SO_PEERCRED way vmd's own gRPC socket already is. execPrivileged now has a third mode, preferred whenever the helper's socket actually exists at call time: root (already privileged, no escalation) → helper (its socket is there) → sudo -n (the pre-existing fallback, still what dev laptops use and what an unmigrated production host falls back to). This removes production's dependency on any sudo flavor being installed at all once vmd-priv.service is deployed and running — the sudo-rs incompatibility above stops being a real blocker on hosts that have migrated, without requiring every host to migrate at once (the socket's mere presence is the switch, not a separate config flag).

Isolation and network¶

• Jailer. Unique uid and gid per VM, chroot, cgroups v2, file-size and open-file limits. Default seccomp filters; no serial console in production. No per-VM network namespace (see below) — the jailer's own uid/gid/chroot isolation doesn't depend on one. Each VM gets its own real, isolated cgroup v2 directory — /sys/fs/cgroup/kapelle/<agent_id> — nested under a shared kapelle parent hierarchy vmd creates once at startup (host.EnsureCgroupParent); the jailer itself only ever appends the per-agent leaf under that already-existing parent, via --cgroup-version 2 --parent-cgroup kapelle --cgroup pids.max=<cap> --cgroup memory.max=<guest RAM + 256 MiB> (confirmed against the real jailer source: --parent-cgroup alone, with no --cgroup property, only does a shared, flat move into the parent for every VM using it — the genuine <parent>/<id> per-VM hierarchy requires at least one real --cgroup property). memory.max bounds the guest's own configured RAM plus a fixed VMM overhead, so a runaway guest can't take down the rest of the host; pids.max bounds the host-side jailer/Firecracker process tree, not the guest's own independent PID space. vmd validates agent_id against a strict charset ([A-Za-z0-9._-]+, no ..) before it ever becomes a cgroup/chroot path segment handed to a privileged binary. KillJail removes the per-VM cgroup directory (bounded retry against a transient EBUSY, logged if it's still non-empty once the window expires) as part of tearing down the chroot.
• Host. Follows Firecracker's production host guide: KSM off, swap off (or secure swap), SMT off on shared hosts, patched microcode and kernel (vmdctl host-check verifies these stay correct after deploy/host/provision.sh --apply provisions a host, design §8's vmd Responsibilities). vmd's own host.MoveKVMPitThreadIntoVMCgroup moves each VM's kvm-pit kernel thread (created by the kernel, post-boot, after Firecracker itself is deprivileged — Firecracker's own docs say this needs an external agent, since nothing else does it) into that same per-VM cgroup, read fresh at call time rather than assumed.
• Network. Each VM's tap lives directly in the host's root network namespace, not its own netns — we never clone VMs, so netns-per-VM isolation had no VM-cloning risk to protect against, and per-tap nftables rules (matched on iifname <tap>) give the same effective isolation for a fraction of the setup cost. Every rule is scoped to one tap by an iifname match; there is no shared state between agents' rules beyond the one nftables table they all live in.
• Egress. The nftables input chain drops by default per tap. A VM may reach only its own host's credential-gateway instance (§9, one iron-proxy process per agent) on its allowed ports; the cloud metadata address (169.254.169.254) and RFC1918 ranges are dropped explicitly, then a catch-all drop for anything else from that tap. DNS (port 53) is not dropped — it's redirected: an nftables DNAT rule sends the guest's DNS queries to its own gateway instance's real DNS listener port, which is not always literally 53 (a dev host without CAP_NET_BIND_SERVICE binds an unprivileged port instead and reports it back to vmd; production binds 53 directly, making the redirect a harmless self-redirect). The gateway resolves names on the VM's behalf and answers only for allowed hosts, so DNS still can't be used to leak data — it's proxied, not left open. vmd's own table is only half the picture on a host that also runs its own firewall (ufw, firewalld, ...): a separate table hooked at the same input priority can independently drop the same packet, invisible to vmd's own rule counters — real, found running this on a real dev workstation. vmd warns at startup if it detects one with no rule covering its configured pool CIDR, but never modifies it; the fix is the operator's (docs/dev-environment.md).
• Inbound. Only worker hosts reach the agent-server port, authenticated with the per-agent session key.
• Snapshots are sensitive. Memory files contain the agent's session key, source code and conversation data, though no outbound credentials. Firecracker only CRC-checks them, so vmd encrypts and authenticates them at rest.

Where it runs¶

Capacity:

• Running agents: limited by host RAM (guest memory, plus at most 5 MiB of VMM overhead each).
• Sleeping agents: limited by disk, about one guest-RAM-sized memory file each, plus data disks.
• Deep-sleeping agents: need only their data disk.

Sandbox provider interface¶

Workers and the controller see sandboxes only through this interface. Firecracker via vmd is the first implementation. Nevia, Aiven's sandbox provider, is planned as the next one. Any provider that can keep a persistent disk per agent and return an agent-server endpoint fits.

class SandboxProvider(Protocol):
    name: str  # "firecracker", later "nevia"
    capabilities: ProviderCapabilities  # sleep, snapshot, persistent_disk, egress_policy

    async def ensure(self, agent: AgentRef, spec: SandboxSpec) -> SandboxInfo: ...
    async def wake(self, agent: AgentRef) -> AgentEndpoint: ...  # url + session key, ready
    async def sleep(self, agent: AgentRef) -> None: ...
    async def destroy(self, agent: AgentRef) -> None: ...
    async def status(self, agent: AgentRef) -> SandboxState: ...

SandboxSpec holds the image, vCPUs, memory, data disk size, egress allowlist, and idle and deep-sleep timeouts. Providers without sleep support report it through capabilities, and the controller keeps those agents running or stops them.

Nevia provider¶

Nevia (Aiven) is the deployment target. Its public API (https://api.nevia.cloud/v1, OpenAPI at /v1/openapi.json, CLI nevia) manages "computers": microVMs running one catalog OCI image as their primary container, with a 20 GiB persistent /data disk, memory-preserving sleep and wake (under a second), checkpoints (memory + overlay + /data) that restore into a new computer in about a second, exec (one-shot and a streamed Connect-over-HTTP/2 session), open egress and, in the current beta, no inbound ingress (published ports are recorded but not enforced) and no custom images. Re-run of the spike on 2026-09-24 (spikes/nevia/FINDINGS.md) confirmed the OpenHands agent-server 1.47.0 runs there unchanged.

How the provider maps onto it:

• Golden checkpoint per role image. One computer is created from the platform default image, the agent venv is installed under /data, and the running agent-server is checkpointed. ensure() restores that checkpoint into a new computer named after the AgentRef. A restore refuses computer-level env (the checkpoint carries the build's own env; the API answers 400 for env, image, ports or size alongside from_checkpoint, confirmed live), so the agent's identity is not baked into the computer at all: wake() passes the session key, secret key and gateway settings as that one exec call's own environment when it runs the bootstrap, and the bootstrap writes them to the agent-server's env file under /data. A new image version is a new checkpoint. A checkpoint is scoped to the computer it was taken from and becomes unrestorable the moment that computer is deleted (confirmed live 2026-09-24, both through the nested restore route and the top-level create with from_checkpoint), so the build computer stays alive, sleeping, as the checkpoint's anchor: one per image version, storage cost only.
• Lifecycle. wake() is Nevia's wake; sleep() is its sleep (memory kept, compute released). There is no cheaper tier below that, so deep sleep is the same as sleep and deep_sleep_after is a no-op for this provider. destroy() cannot keep a checkpoint as the backup for the reason above; instead it puts the computer to sleep and renames it archived-<agent>-<timestamp>, which keeps the disk recoverable at storage cost and counts against the workspace's computer cap, and skip_backup=True deletes outright. Exporting /data to our own object storage before delete is the follow-up that removes the cap pressure. All four capabilities flags except egress_policy are true.
• Endpoint. Until Nevia issues the HTTPS origin the spec already describes for a published port, the worker reaches the agent-server through the exec stream: wake() mints a stream URL, runs a small TCP relay inside the computer, and exposes it as a loopback listener on the worker host, so AgentEndpoint.url is a plain http://127.0.0.1:<port> exactly as with Firecracker. The signed stream URL is a credential and is held only by the executor's session.
• Guest bootstrap. No MMDS, systemd or read-only root: a supervisor script in the golden image reads the same fields from the environment, writes the agent-server env file and CA bundle, and starts the agent-server; the provider re-runs it through exec when a restored computer comes up without it. The Nevia image provides the same /workspace layout as the Firecracker image (/workspace/project, /workspace/.kapelle-skills) through a link /workspace -> /data/workspace, made at image build and re-checked by the bootstrap on every wake, so the worker keeps one guest layout for every provider.
• API credential. Nevia's API takes a Zitadel access token (JWT) that lives 15 minutes. The public API has no service account, API key or automation token in v1 ("Service accounts are deferred"), and the documented way to get a bearer is nevia login once, then nevia auth access-token, which prints the current token and refreshes the CLI's own session, so a worker that runs for hours cannot hold one static bearer. The provider authenticates through a token source. KAPELLE_NEVIA_TOKEN is a bearer used as is (an automation bearer that something else keeps fresh; a 401 fails with a message saying the token is expired or invalid). KAPELLE_NEVIA_TOKEN_COMMAND is a command whose stdout is the token, typically nevia auth access-token (an absolute path is the safe value, the worker's PATH must find it otherwise): it runs without a shell, the token is cached and read for its exp claim (timing only, no signature check), and the command runs again 60 s before expiry and once after a 401. The worker never reads, parses or writes the CLI's credentials file; the CLI owns it. Setting both variables is a startup error and there is no default command. Every API call, the exec-stream URL mint and the backup client go through the source. The command source is a development bridge, not the production design: it acts on Nevia as whoever is logged in to that CLI, in every workspace that user can reach, and it is subject to that session's own lifetime (an expired session needs nevia login again). The recommended production setup is a machine identity, once Nevia offers one. Confirmed live (stage 3b): an existing avn_... Aiven Cloud API key is rejected outright for this API (401, ERR_JWS_INVALID, "Invalid Compact JWS") -- not merely absent, a different token format entirely; only a Zitadel JWT bearer works.
• Credential gateway. A Nevia computer has no tap of ours and open egress, so the per-tap identity and nftables forced path of §9 do not apply. Probed live on 2026-09-28 (spikes/nevia/FINDINGS.md): Nevia computers cannot reach each other and cannot be reached from outside, so an agent cannot dial a credgwd listener directly. The agent's proxy traffic therefore travels through the exec stream the worker already holds (the reverse relay). wake() starts images/nevia/kapelle-nevia-relay in the computer over a second exec stream; it listens on 127.0.0.1:18080, which is the agent's HTTP_PROXY/HTTPS_PROXY (plain HTTP, no credential in the computer at all), and carries every connection as multiplexed frames on its stdout and stdin. The worker demultiplexes them and dials the agent's own credgw instance, at the private tunnel listener StartInstanceResponse.tunnel_listen (loopback of the host credgwd runs on, not the shared public listener), so the instance is chosen by which stream the bytes arrive on and the same iron-proxy policy applies. The frame protocol (nevia_relay_protocol.py) gives each connection a per-direction credit window (256 KiB) so one slow connection never blocks the others, with half-close, a PING every 10 s that the relay answers, and an idle exit in the computer after 30 s without a frame. Nevia can stall an exec stream without closing it, so the answer matters: a stream from which nothing at all arrives for 30 s is aborted and replaced (only for a relay that advertises pong in READY, so an older image keeps working). Both relays (the forward relay to the agent-server and this reverse one, with its pings and silence watchdog) run on their own thread with their own event loop, because the worker's main loop is not always free: the OpenHands SDK makes synchronous HTTP calls to the agent's endpoint from the loop that would have to serve that endpoint, and a relay served by a blocked loop deadlocks (found live: 10 minutes without a model call, then the in-computer relay exited on its idle rule). The main loop mints and refreshes exec-stream URLs and publishes them to the relay thread, which never touches the Nevia API client or the token source. The stream URL is a bearer that cuts a session still open at its expires_at, and closing a stream does not stop the process in the computer, so renewal is make-before-break: shortly before expiry a second stream starts a second relay bound to the same port (SO_REUSEPORT), the first is told to DRAIN and exits when its last connection ends, and a stream that ends by itself is restarted with back-off. KAPELLE_NEVIA_CREDGW_MODE=public keeps the earlier mode (the agent dials the shared public TLS listener with a per-agent client token, a placeholder never a real secret), which also serves as the per-agent fallback when credgwd returns no tunnel_listen. HTTPS destinations such as github.com are intercepted, so the computer must trust credgwd's interception CA: the provider fetches it once (GetCACertificate, a remote instance's own ca_cert_pem is only the public listener's certificate) and hands the guest a bundle of that CA, plus the listener certificate in public mode; if credgwd cannot supply it, wake() fails instead of starting an agent that cannot reach GitHub. The allowlist still governs what gets a credential; direct internet access from the agent is not blocked, which the security section records as the accepted difference from the Firecracker host.

Existing managers considered¶

Cloud Hypervisor (pause, snapshot, restore and live migration) is the fallback VMM if Firecracker's snapshot limits prove too tight.

Credentials¶

Agent VMs hold no outbound credentials. A credential gateway on each VM host, outside every VM, is the only place a VM can send traffic. For each request it works out which agent sent it from the VM's network interface, checks that agent's policy, adds the right credential and forwards it. Long-lived secrets stay in OpenBao; the gateway only holds short-lived tokens.

flowchart LR
    VM["Agent microVM<br/>payments/coder-1<br/>placeholder credentials only<br/><i>nftables: only the gateway is reachable</i>"]
    CG["Credential gateway<br/>identify agent by its tap<br/>check policy: host, path, repo<br/>strip, then add credential<br/>write audit record"]
    BAO[("OpenBao")]
    AUDIT[("audit log")]
    LLM["LiteLLM · worker MCP"]
    GH["GitHub API · git"]
    REG["Package registries"]

    VM -->|"no credentials from its tap"| CG
    BAO -->|"short-lived tokens"| CG
    CG --> AUDIT
    CG -->|"+ credential"| LLM
    CG -->|"+ credential"| GH
    CG -->|"+ credential"| REG

The agent can make allowed requests but never sees a real credential. Requests to anything outside its policy are refused at the gateway, and direct connections are dropped by the host firewall.

What the gateway adds¶

How it works¶

• One gateway instance per agent, not one shared gateway. There is no per-VM network namespace — vmd never clones network namespaces, the only reason Firecracker's own docs recommend one (2026-09-13 network-model decision). Every agent's tap lives directly in the host's root namespace with its own /30; vmd calls credgwd's StartInstance (with the agent's AgentRef and the host-side address of that /30) as it creates the agent's network slot, and credgwd starts that agent's OWN iron-proxy process bound explicitly to that address — never a wildcard, since every instance on the host shares the same namespace. So identity is never inferred from a source address at request time: it comes from which instance a request reaches, decided once at StartInstance time. vmd calls Reload/Stop the same way as the agent's slot changes or is torn down.
• nftables is mandatory, not a hardening extra. Since every instance shares the host's root namespace, nftables (keyed by the agent's own tap interface, not by IP) is the ONLY thing stopping one agent's traffic from reaching another agent's gateway instance, the host's other addresses, or anything bypassing the gateway entirely — without it, "one instance per agent" would isolate nothing. A host that can't run the nftables base table isn't a safe place to run agents.
• Forced path. The host firewall allows a VM only the gateway's ports, so a direct connection fails. The VM's resolver points at the gateway, which answers only for allowed hosts.
• TLS. Each host's gateway has its own CA. Its certificate is in the VM's trust store, and SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS, HTTPS_PROXY and HTTP_PROXY are set through MMDS. HTTPS_PROXY/HTTP_PROXY point at the gateway's dedicated CONNECT/SOCKS5 tunnel listener specifically (StartInstanceResponse.gateway_tunnel_url, proxy.tunnel_listen — backlog task 42 run5), not its plain HTTP listener: confirmed via strace against a real iron-proxy binary that the plain listener has no CONNECT handling at all (it forwards a CONNECT request verbatim to a fresh outbound connection, which a real TLS-only upstream rejects outright), so only the dedicated tunnel listener actually hijacks a CONNECT, mints a leaf cert from the host CA, and MITMs it — evaluating the same allowlist/secrets transforms as any other request. The gateway issues certificates only for allowed hosts.
• Internal services without interception. LiteLLM and the MCP tools are plain HTTP endpoints on the gateway, which uses TLS for the onward connection.
• Guest-facing URLs are a separate concern from the allow-rule host. credgwd's -llm-host/-mcp-host flags stay bare host, no port — they set the llm/mcp destinations' allow-rule Host, and iron-proxy's allowlist matcher strips the request's own port before comparing, so a host:port value there silently 403s every request instead of matching (found for real through a microVM guest). The real, dialable port for the agent's own URL is a separate pair of flags, -llm-port/-mcp-port; the result reaches the agent as vmd's AgentEndpoint.gateway_llm_url/gateway_mcp_url. gateway_llm_url is the bare LiteLLM base with no trailing slash (the OpenHands SDK's own leading-slash path join would otherwise double up into a literal // that a real LiteLLM 404s on); gateway_mcp_url always ends in /mcp, the worker's own MCP mount root, onto which the worker joins its per-conversation path. See credgw/README.md's "Guest-facing URL contract" section for the full reasoning, and its "CONNECT/TLS-intercepted destinations" section for gateway_tunnel_url specifically.
• Strip, then add. The gateway removes Authorization, x-api-key and similar headers from the agent's request before adding its own, so nothing the agent supplies passes through.
• git. Remotes use HTTPS (url."https://github.com/".insteadOf git@github.com:), and SSH egress is blocked. The added header is the same one actions/checkout sets through http.extraheader.

Where secrets live¶

• OpenBao (MPL-2.0, 2.6.2) holds:
• the GitHub App private key
• per-agent LiteLLM virtual keys
• registry tokens
• surface credentials (Slack, Linear, Jira, GitHub webhook secrets)
• vmd's snapshot encryption keys
• GitHub tokens are minted when needed: installation tokens limited to the team's repos and permissions, valid for 1 hour. The controller mints them (Controller.MintGitHubToken, contracts/proto/controller.proto) rather than an OpenBao plugin — never stored in OpenBao at rest, since a fresh mint is as cheap as a read. The gateway caches each token and refreshes it before expiry (creds.Store, credgw/creds/creds.go).
• LiteLLM keys never leave OpenBao and the gateway. The VM only has a placeholder, so keys can be rotated at any time without touching conversations.
• Gateway access. The gateway authenticates to OpenBao as its host, and may read only credentials for agents placed on that host.
• Destination secrets and credgw. A Console destination that needs a credential names it as inject_secret: secret:<ref-name>; the controller writes the value to secret/data/kapelle/console/destinations/<ref-name> (field value; integration secrets under console/integrations/ are separate and never readable by credgw) and the gateway reads it from there. That path is deployment-level, not per host: every host's credgwd can read every destination secret, not only those of agents placed on it. The accepted tradeoff for this release is a bigger blast radius if one host's gateway is compromised, in exchange for one copy of each secret and no per-agent fan-out of values. The gateway's AppRole policy (deploy/compose/openbao/policies/credgw.hcl) is read-only on secret/data/kapelle/console/destinations/* and on secret/data/kapelle/teams/+/mcp/* (a team's external MCP server secret, inject_secret: mcp:<destination>), nothing else under a team. A per-agent copy written at agent start, keeping the host-scoped read, is a listed follow-up (backlog task 00cb689e). The value never reaches the guest: it is swapped in by the gateway for the placeholder on requests to that destination only.
• Left in the VM: OH_SESSION_API_KEYS_0, which only grants access to that VM itself, and OH_SECRET_KEY, which encrypts conversation state on its disk.

Policy and audit¶

• Per role profile. Allowed hosts, HTTP methods and paths, deny by default (credgw/policy/roles/<role>.yaml, one file per role — planner/coder/reviewer so far). For example, a reviewer can read pull requests and comment, but not push (methods: [GET] on its github-api destination). The coder's own github-api destination is scoped to pulls/issues/search/user only, not the repo's contents/git-refs/releases/settings — the platform pushes the branch (a separate github-git destination, gated by git_push_ref_globs), and opening the PR itself is everything github-api needs (backlog task 42; §11's "The coder's PR, opened deterministically").
• Where a policy comes from, and egress modes. credgwd -egress-source=file (the default) reads the role files above. -egress-source=controller asks Controller.GetEgressPolicy for the policy generated from the Console's records (an agent definition's egress, replaced by a team's per-role override), and falls back to the role file only for a role the store has no agent definition for. A controller that cannot be reached at an agent's first start fails that start; a later start serves the last good policy. A role's egress mode is allowlist (only the named destinations, plus the platform's own llm, mcp and otel) or allow_all: one extra * allow rule, so any public host is reachable through the gateway, still with the deny CIDRs, still with named destinations' credentials injected only on their own hosts, and with no secret in the guest. Allow-all keeps CONNECT to any port in this release. On Nevia the egress mode governs traffic through the credential gateway only; the computer's own network stays open. Firecracker enforces it at the network.
• Package registries, pass-through. Every role also gets deb.debian.org/security.debian.org (apt), pypi.org/files.pythonhosted.org (pip) and registry.npmjs.org (npm) destinations with no inject_secret at all (backlog task 08d11d35, paired with the guest's agent user getting passwordless sudo — task 4c5003e9 — so an agent may install what a task needs). Pass-through means exactly that: the allowlist still bounds egress to these hosts and audits every request the same as any other destination, but nothing sensitive is ever attached to one of these requests, since there's no credential to inject in the first place.
• Branch scope, enforced deterministically. Agents push only to agent/<team>/*. This is NOT the LLM-judged check the original design considered (a judge transform is a per-push model call, non-deterministic, and a path for a model-provider credential to end up inside the gateway — rejected 2026-09-13): a small refcheck gRPC service, one per agent instance that needs it, parses a git-receive-pack request's pkt-line ref-update commands directly and rejects the push unless every ref matches the role's git_push_ref_globs, wired in as one of iron-proxy's own transforms. GitHub-side branch rulesets remain a second, independent layer, not a substitute.
• Audit. iron-proxy exports its own structured JSON audit trail as OpenTelemetry logs natively — the gateway doesn't reimplement this — with host/method/path/action/status_code/duration_ms per request, plus OTEL_RESOURCE_ATTRIBUTES (agent id, team, task id) set on each instance's own environment so every exported record carries who it was.
• Placeholder-header contract. The VM only ever sends the literal "placeholder" as the LLM api_key, the MCP/github-api Authorization: Bearer value, and the password half of github-git's Authorization: Basic base64(<any-username>:placeholder) value — never a real credential, never a value it derives itself. iron-proxy's secrets transform substring-replaces that literal wherever it finds it in the client's own Authorization header (base64-decoding first for Basic), swapping in the real credential WITHIN whichever scheme the client sent — it never chooses or rewrites Bearer vs. Basic itself, so the client must send whichever scheme the real upstream expects (Bearer for GitHub's REST API, Basic for GitHub's git smart-HTTP endpoints — confirmed live, backlog task 42: a Bearer-scheme request reached GitHub with the real installation token already swapped in and still got a plain 401, since GitHub's git-over-HTTPS never accepts Bearer). See credgw/README.md's own contract section for the exact call sites on both sides.
• MCP routing headers. Every role's mcp destination header_allowlist must include Mcp-Session-Id, Mcp-Protocol-Version, Mcp-Method and Mcp-Name — the MCP streamable-HTTP transport's own required headers. header_allowlist is a default-deny request-header filter, so without these four the real MCP server rejects every call after the initial initialize (Mcp-Name specifically only matters for a tool-CALL-shaped request, easy to miss if a fix only re-verifies initialize). A Go-side test (credgw/policy/policy_test.go) is a static regression guard; services/worker/tests/mcp/test_header_allowlist_matches_sdk.py derives the true required set straight from the installed mcp SDK's own header constants each run, so a future SDK version adding a header fails that test first, not silently this one.

Secret hygiene¶

• Redaction at the log boundary, both languages. credgwd's own slog output goes through credgw/redact.Handler; kapelle_contracts.redact (contracts/python) is the same two-layer design in Python, independently implemented since the two runtimes don't share code elsewhere either. Layer one is known-shape prefix patterns needing no registration (a LiteLLM key's sk- prefix, GitHub's gh[oprsu]_/github_pat_ prefixes, OpenBao's hvs./s./b.-prefixed token shapes); layer two is an exact-value registry (RegisterSecret/register_secret) for secrets with no distinctive shape, such as credgwd's own minted management API key or an OpenBao AppRole role-id/secret-id. Wrapping the logging handler itself means even an error value wrapping a raw upstream response body gets redacted, not just a call site that happened to pass a plain string.
• Leak scanning. just scan-secrets (scripts/scan_leaks.py, over kapelle_contracts.leak_scan.scan_paths) audits real files/directories — an .env file, or OpenBao-backed agent secrets read live (--openbao-agent/--openbao-path/--openbao-list-agents) — for those values leaked into logs or a real microVM run's leftover work directory, printing file:line plus redacted context, never the raw value, on a hit. just nightly-scan-secrets runs this as nightly CI's own post-run credential-hygiene step, against the containerized control plane's logs plus any native vmd/credgwd/iron-proxy logs a real microVM phase left behind.

Implementation options¶

Limits¶

• Only HTTP(S) header or Basic credentials can be added. Request signing such as AWS SigV4, and SDKs that fetch their own OAuth tokens, need the gateway to do the signing.
• Some tools break. Tools that pin certificates or ship their own CA bundle fail, safely, because direct connections are blocked.
• The agent can still use credentials. It can't read them, but it can make any request its policy allows, so narrow policies and the audit log carry the weight.
• Non-HTTP secrets can't be hidden. A database password is one example. Use disposable services inside the VM for tests instead.
• The gateway is a high-value target. It holds live short-lived tokens for every agent on its host. It stays small, reachable only from its host's VMs, and has no long-lived keys.

Models¶

Only the LiteLLM proxy holds provider credentials. Agents use model aliases and LiteLLM virtual keys, so moving a role between a self-hosted model and a commercial API is a change to the alias.

• Aliases. Role templates name aliases such as local-coder, local-large or frontier, and the proxy maps each to a vLLM deployment or a provider. Since 2026-09-24 the Aiven AI Gateway is the only provider: the aliases (frontier, frontier-large, frontier-xlarge, fast, open-coder, plus local-coder/local-large kept as compatibility names for the same models) come from deploy/compose/litellm/aliases.yaml -- the source of truth for which alias points at which underlying model, not config.yaml itself, which just litellm-sync-aliases regenerates from it (docs/dev-environment.md's "Model providers" section).
• Team budgets. /team create calls /team/new with max_budget and budget_duration.
• Empty answers are errors. A proxy callback (deploy/compose/litellm/empty_response_guard.py) turns an upstream answer with zero completion tokens and no content (no non-blank text, refusal, tool call or reasoning summary; an output item holding only blank text does not count) into a 503 naming the model, for /v1/chat/completions and /v1/responses, streaming or not (a stream is held back only until the answer starts). The agent SDK retries a 503 with backoff, where a silent empty 200 used to reach the agent as a success and end in its stuck detector. LiteLLM's own num_retries does not apply: the callback runs after the router. On LiteLLM v1.103.0 and later (traced to BerriAI/litellm#40243), a raw, genuinely-incrementally-streamed /v1/responses caller gets 200 with the failure embedded as an event: response.failed SSE frame naming the model instead of a real 503 -- Chat Completions is unaffected. This codebase's own callers DO reach that shape (backlog task 3408802f, reopening 02115ab8's earlier belief that none did): once the proxy answers /v1/model/info for an alias, as the real config.yaml does, the SDK's own client genuinely requests streaming for the Responses API too. litellm's own client converts the response.failed frame into a MidStreamFallbackError/ServiceUnavailableError before it reaches openhands-sdk -- still in LLM_RETRY_EXCEPTIONS, so the SDK retries it exactly like the clean 503 on older versions; proven live against the real config.yaml and a real stub upstream on both v1.100.1 and v1.103.0, both a transient empty answer (recovers after one retry) and a persistently empty one (all retries exhausted, then a loud failure). For Chat Completions specifically, one guard-triggered 503 costs 3 real upstream calls per SDK-visible attempt, not 1 (backlog task 34baedc7): the openai-python client inside litellm's own provider handler retries the 503 itself, on its own default max_retries=2, unrelated to and unreachable from anything this codebase configures -- accepted because an empty answer is rare, costs no completion tokens, and the client's own two quick repeats reach a good answer sooner than the SDK's own first retry, whose backoff starts near 8s, would.
• Per-agent keys. Created with /key/generate, setting team_id, models, max_budget, budget_duration, rpm_limit and tpm_limit, and capped by upperbound_key_generate_params. The credential gateway adds the key to each request. The VM only has a placeholder, so keys can be rotated at any time (§9).
• vLLM. Tool calling must be enabled explicitly (--enable-auto-tool-choice and the model's --tool-call-parser). A model is admitted for a role only after passing the evaluation suite (§18).
• Enterprise-only LiteLLM features we don't use: SSO beyond 5 users, audit logs, automatic key rotation, tag-based and per-model key budgets, soft budget alerts, per-key guardrails.

Agent-to-agent communication¶

Every role is both an A2A server (its executor) and an A2A client (its MCP tools), so any agent can delegate to any role in its team. An agent waiting on a reply ends its run and may fall asleep. The reply arrives as an ordinary queued message for that agent, and the worker that picks it up wakes the agent.

sequenceDiagram
    participant A as A · agent microVM
    participant AE as A · executor
    participant N as NATS JetStream
    participant B as Agent B

    A->>AE: send_task (MCP tool)
    AE->>N: SendMessage → B.rpc
    AE-->>A: taskId · run finishes
    N->>B: deliver · ack at SUBMITTED
    B->>N: INPUT_REQUIRED event
    Note over A: idle, may sleep
    N-->>AE: reply → A.rpc (durable)
    AE-->>A: wake · send_message · run()
    A->>AE: reply, same taskId
    AE->>N: SendMessage
    N->>B: deliver · B wakes and resumes
    B->>N: COMPLETED + artifact
    Note over A: idle
    N-->>AE: reply → A.rpc (durable)
    AE-->>A: wake · send_message · run()

Delegation with a clarifying question. The "reply → A.rpc" and "wake · send_message · run()" steps are the wake-up loop: B's worker turns the event into a SendMessage on A's rpc queue, and whichever worker picks it up wakes A's microVM if it is asleep and resumes A's conversation. The reply survives worker crashes because it is a queued message, not a subscription held in memory. Subject names are shortened; full names are in §6.

Rules¶

• Durable replies. A reply to a delegation is a SendMessage on the requester's rpc subject (§6), carrying the delegated task id, so the requester's executor can tell a reply from new work. Surface adapters waiting for results use the same mechanism instead of in-memory subscriptions.
• One run at a time per conversation. Runs for a given contextId and role are serialized. An agent's max_concurrent_runs caps parallel work items.
• Busy agents. Messages queue as new tasks. Injecting into a running run is out of scope. While a task waits for a free replica or a waking agent, the surface shows it as queued or waking (§13).
• Hop limit. A delegation depth in A2A metadata, plus max_iteration_per_run and key budgets, stops loops.
• Broadcast. Plain NATS publish/subscribe outside A2A, or the lead sends the work out.
• Deterministic pipeline hops. A template can declare pipeline: {role: next_role} (§12): when a role's run completes with a pushed branch and it has a configured next hop, the executor calls send_task to that role itself, unless the model's own conversation already delegated to it. Whether a delegation chain actually advances then no longer depends entirely on the model choosing to call the tool.
• The coder's PR, opened deterministically. Right after a successful push, the worker (not the model) opens or reuses the pull request for that branch — a real GitHub API round trip run via the guest's own execute_bash, so it flows through credgw's pulls/issues-only github-api destination like any other agent request (backlog task 42, executor.py's _open_pull_request_after_run). This happens before the pipeline hop above, so a configured next-hop role always sees a real PR URL in the outcome it receives. The title is the first line of the task as delegated (at most 72 characters, cut at a word boundary; pr_title.py), not the agent's final message; the body is the final message, a one-line task footer, then Closes #N. The commit the worker makes for changes the agent left uncommitted is named the same way: chore(agent): <task title> with the task id in the body.

Teams¶

A team is a record in Postgres: its roles, the model alias and VM profile for each role, repo, budget, linked surfaces, and its agents. Agents are created when a role first gets work, sleep when idle, and are added as extra replicas when a role's queue backs up. Workers stay shared and stateless.

Creating a team¶

/team create payments --template feature-team --repo git@github.com:org/payments.git
/team link payments slack C0123456789
/team link payments linear PAY
/team archive payments

--template may be left out only where the deployment names a default template (KAPELLE_DEFAULT_TEMPLATE; the dev stack sets feature-team, the Nevia control plane feature-team-nevia): a template decides where the agents run and what they may cost, so the platform never chooses one in silence. With the variable unset, /team create without --template is refused with the usage line and the templates that exist; a name that matches no template file is answered the same way.

The team controller:

1. Creates the team's home room: a public Mattermost channel by default, or a Slack channel with --home slack (KAPELLE_DEFAULT_HOME_SURFACE changes the default; the creating person is added to a Mattermost channel when their Mattermost account is linked). Every step of the team's work is posted to the home room; other linked surfaces get only their own work items.
2. Creates the LiteLLM team and its budget.
3. Writes the team record.
4. Registers Agent Cards (payments.planner, payments.coder, …).
5. Records surface links.

No VM is created until a role receives work. /team archive cancels open tasks, removes cards, stops routing, and destroys the team's agents after their data disks are archived.

Templates¶

# teams/templates/feature-team.yaml  (model values are LiteLLM aliases)
lead: planner
roles:
  planner:  { agent: planner,  model: frontier,    vm: small,  replicas: { max: 1 }, concurrent_runs: 3 }
  coder:    { agent: coder,    model: local-coder, vm: medium, replicas: { max: 3 }, concurrent_runs: 1 }
  reviewer: { agent: reviewer, model: frontier,    vm: small,  replicas: { max: 1 }, concurrent_runs: 2 }
sleep:
  idle_after: 10m
  deep_sleep_after: 24h
budget_usd_per_day: 50
isolation: shared            # or: dedicated workers and NATS account
rework_rounds: 2             # optional, default 2: review-and-fix cycles per work item (Team flow)
pipeline:                    # optional: deterministic hops (§7)
  coder: reviewer
  reviewer: planner

Each agent: value points to a shared role definition (system prompt, tools, subagents, confirmation policy). The role's tools: list also governs the worker's own MCP tools (§7): a role is offered send_task, get_task, ask_requester, memory_* and the rest only if its definition names them (a role file with no tools: list is unrestricted), enforced by all three runtimes. The shipped roles do not name get_task: a lead that has delegated ends its turn and is woken by the reply, it never polls. Each vm: value names a SandboxSpec profile (image, vCPUs, memory, disk, egress allowlist). Roles default to NeverConfirm because the microVM and the gateway policy are the boundary; roles that can take costly or hard-to-undo actions (for example one allowed to open PRs against release branches) use ConfirmRisky, which surfaces as INPUT_REQUIRED for a person to approve.

See docs/team-isolation.md for what isolation: dedicated actually does today (worker pinning only, backlog task 90ad7902) versus what it doesn't yet (a dedicated NATS account, backlog task d74d9c67) -- including the operator steps to start a pinned worker and the hard constraint that a pinned and a shared-mode consumer can't share one A2A_RPC stream.

Agents and work items¶

• Sticky routing. A work item's first task picks the least-loaded awake replica, then a sleeping one, then a new one up to max. Later tasks on the same contextId go to that replica.
• The lead is a bottleneck by default. Every work item starts with the lead, so a team's throughput is bounded by the lead's replicas.max × concurrent_runs, and its VM holds a worktree per work item. Templates may set replicas.max above 1 for the lead; sticky routing keeps each work item with one replica, and the lead keeps no cross-item state that would be lost by spreading items across replicas.
• Worktrees. Each work item has its own git worktree and branch in the agent's data disk. Coder and reviewer are separate agents with separate disks, and exchange work through branches.
• Blast radius. An agent VM holds several work items for one team role, never work from another team.
• Budgets. Enforced by LiteLLM per team and per agent key. The controller posts a notice in the team room when a budget runs out.

Team flow¶

The team is a hybrid: one lead owns each work item, deterministic hops carry the main path, and roles may still talk to each other directly. Written out, the rules the pieces above imply:

• The lead owns the work item. Every work item, from any surface, starts as a task on the template's lead role. The lead is the only role that speaks to the person: it asks clarifying questions, reports progress and declares the item done. Other roles reach the person only through the lead, except for a risky-action approval, which goes to the room directly (§7).
• The main path is configured, not decided. pipeline: {coder: reviewer, reviewer: planner} fixes the handoff chain: a coder's pushed branch always goes to review, and the review verdict always returns to the lead. The worker performs these hops and opens the pull request; the model never has to remember to.
• Direct delegation stays bounded. Any role may send_task to any other role and ask_requester up its own chain, which is how the lead delegates and how a reviewer sends a fix back to the coder. Delegation depth, per-turn iteration budgets and key budgets bound the loops; a template's team-level rework_rounds (an integer >= 0, default 2; /team show prints it, /team set <team> rework_rounds <n> changes it, /team refresh leaves it alone) bounds how many review-and-fix cycles one work item may take before the lead has to report the state to the person instead of retrying.
• What counts as a rework cycle. One reviewer verdict that sends the work back to the coder: either the reviewer's own send_task to a role other than the lead (from a role whose configured next hop is the lead), or a final message whose first line is CHANGES REQUESTED. Each verdict is recorded once per work item (contextId) in the worker's Postgres bookkeeping, keyed by the reviewer's task id, so the count survives a worker restart and the two detections never double count. Verdict n goes back to the coder while n <= rework_rounds. The next one does not: the platform appends a rework-limit note to that verdict (it reaches the lead as the reply), and send_task refuses every further delegation in that work item except a report to the lead. The lead then reports what was done, the branch and PR, and what the reviewer still objects to, and the work item stops. The cap bounds autonomous loops, not the person: a message from a person to the lead (a task from a surface, or a follow-up that resumes one) starts the count over and lifts the refusal, while a role's durable reply never does; the earlier verdicts stay recorded for audit.
• Done means the lead said so. A work item is complete when the lead's task completes after the pipeline returned to it: the pull request exists, the reviewer approved, and the lead's final message names the branch, the PR and what was left open. The platform records the branch and PR of the coder's latest push per work item and, when a completed lead message omits either, appends them to it rather than failing the task; what was left open is the lead's to say. That message becomes the outcome memory entry (§14).
• Waiting costs nothing. A lead that has delegated ends its run and its VM sleeps; a durable reply wakes it. A blocked delegate (no tools, no credential, a failing environment) reports the block instead of retrying; the lead retries once, then escalates to the room with the reason, as it did in practice when a coder found itself without a shell.
• A durable reply is never silently discarded, even if its own requester already completed. Backlog task c3e7346b, real incident: a coder's completion reply, deferred because the planner (lead) was still WORKING, was dropped outright once the planner reached COMPLETED first — real, needed information (the branch, the PR) never reached the reviewer, and the delegation chain broke. Reopening the requester's own finished task isn't an option (the a2a-sdk refuses to resume an already-terminal task), so instead: a reply whose requester is INPUT_REQUIRED still resumes that exact task, and a reply whose requester has already reached a terminal state — or was never tracked at all — lands as a brand-new task on the requester's own role rather than being withheld. A follow-up real run then showed landing this way isn't quite enough on its own for a role finishing a configured pipeline hop (reviewer, for pipeline: {coder: reviewer}): its technical requester is coder, not the lead, so a bare landed reply addressed to coder arrived as a brand-new, context-free task with none of coder's own history — and failed outright. Fixed two ways: such a completion now routes to the lead instead ("the lead owns the work item" — GetTeamRole's lead_role, redirected to whenever the requester role's own pipeline_next_role matches the finishing role), landing on the lead's own most recent task for the work item; and any landed (never resumed) reply now carries the requester's own conversation_id/origin so it continues with real history instead of starting blank. See contracts/a2a-nats.md's "Reply rule" for the full mechanics.
• A pipeline hop replaces the completion reply to its own role. Backlog task 3e69270c, real incident: when the reviewer sent work back and the coder's rework task completed, the reviewer received the coder's push twice — once as the configured hop and once as the completion reply to the task's requester (the reviewer) — reviewed it twice and spent two rework cycles. The executor knows when it is about to perform the hop and announces it on the completion; the reply to that same role is then not sent (failed, cancelled and question replies are unaffected), and if the hop then fails the reply is sent after all. contracts/a2a-nats.md's "Reply rule" has the mechanics, including the one accepted duplicate after a worker crash between completion and hop.
• A delegate's question is answered by the platform, not by a tool. Backlog task cdbdb511, real incident: a reviewer that asked its requester a question got the question as a durable reply, the requester's turn answered it, and the answer stayed in that turn's own task — the paused reviewer waited forever. Now a turn started by a delegate's question (the incoming message carries kapelle.reply_to_task for a task paused on a question, never a confirmation) owes that task an answer. When the turn ends COMPLETED, the worker sends its final message to the delegate as a SendMessage with the delegate's own taskId, so the paused task resumes exactly as in the sequence diagram; if the answerer asks upward first (for the lead, the person in the room) the pending target is kept in the worker's Postgres and forwarded once that task completes; if it ends FAILED/CANCELED the delegate gets a platform note to proceed on its own judgement and state its assumptions. At most three questions per delegated task are routed to an agent; the fourth is not routed and the delegate is resumed with a note that the clarification limit is reached. The question message tells the receiving role that its final message is the answer and to use ask_requester when only the person can answer. See contracts/a2a-nats.md's "Reply rule".
• Parallel work is per work item. Sticky routing keeps one work item on one replica of each role; extra replicas serve other work items. A single work item is never split between two coders by the platform; if a lead wants that, it delegates two explicit sub-tasks and merges their branches itself.

Cost and attribution¶

Every agent's own LiteLLM virtual key is minted per (team, role, replica) at EnsureAgent time and written to OpenBao alongside its other secrets (Credentials) — every real completion an agent makes carries that key, so LiteLLM's own spend records attribute back to a real team and agent, not a shared key. The dev-only fake credential gateway (FakeProvider(use_real_agent_server=True): a real agent-server, no microVM) mirrors this rather than shortcutting it: it resolves the same OpenBao-minted key per agent instead of injecting one flat key for everyone, so cost attribution holds in dev too, not only on real Firecracker. just spend (scripts/spend_report.py) reports spend for a window by model, by agent key, and by team — the report this attribution exists to feed; a caller with no OpenBao credentials configured falls back to a single shared key with a warning, a visible degradation rather than a silent one.

The same per-work-item task-creation metric that lets a runaway delegation loop be caught before it burns real spend (kapelle_tasks_per_work_item_total, Observability) uses the same kind of caller-scoped labeling — team/role/work-item, not a global aggregate — for the same reason: cost and safety both need to be traced back to a specific team and a specific unit of work, not just totaled.

Surfaces¶

The surface gateway turns each surface into A2A:

• a work item becomes a contextId
• a human message becomes SendMessage
• an agent question is INPUT_REQUIRED
• results are artifacts

Replies go back to the surface the work came from, and every step is also posted in the team's home room. The gateway drives the human-facing round trip for the lead's own task directly, but a work item's whole conversation crosses several roles (§11), and most of that never touches the gateway's own request — the lead delegates and moves on. To still see it, the gateway also runs one durable NATS consumer per team on the team.<team>.tasks.*.events fan-out, tags each event with the role that produced it, and matches it back to the open work item by contextId; a delegated role's activity reaches the room this way, not by the gateway watching that role's task directly. A work item closes once the lead's own context has no task left open — the same terminal states this consumer already watches for.

Planned, not built yet (backlog task 03f34510): team members show up as bots of their own — coder, reviewer, each posting under its own account — on Mattermost first, then Slack; a role without a bot falls back to today's shape, the one surface bot below with the role named in the text.

Slack¶

• Events: Socket Mode (apps.connections.open, app token with connections:write), so no public endpoint is needed. Subscribe to app_mention and message.channels.
• Posting: chat.postMessage with thread_ts, about 1 message per second per channel. Streaming uses chat.startStream, chat.appendStream and chat.stopStream.
• Status: agents.sessions.setStatus takes processing, suspended and closed. agent_session_stopped maps to CancelTask. Some features need a paid plan.
• Commands: slash commands don't work inside threads, so use them for /team and for starting new work; use mentions in threads.

Mattermost¶

• Events: the bot's WebSocket (/api/v4/websocket, authentication_challenge), so no public endpoint is needed. posted, channel_created and user_added are handled; post_edited is ignored. Each event carries a sequence number: a gap, a reconnect (backoff capped at 30 s) or a gateway restart triggers a catch-up that fetches the posts made since the newest one delivered (GET /channels/{id}/posts?since=, bounded by a configurable age, 24 hours by default), so nothing is lost; the checkpoint lives in Postgres.
• Work item: any top-level post by a linked person in a team's channel starts one; its thread carries the rest. A reply in a thread that is not a work item is ignored. The bot's own posts and those of other bots are never acted on.
• Posting: POST /api/v4/posts with root_id, split below MaxPostSize. A role's status (working, done, a verdict, with its branch and pull request) is a new post each time it changes, batched to one per role per flush interval, so the thread reads as a sequence and a post is never edited (decided 2026-10-02 after the first real session: an edited post hid that the coder had gone back to work after a "changes requested"). The activity summary is one post per thread edited in place (PUT /posts/{id}/patch) until it would exceed the limit.
• Commands: the client rejects unknown slash commands, so there is one custom slash command, /kapelle <command> (/kapelle status, /kapelle team create demo --home mattermost), which Mattermost delivers to POST /webhooks/mattermost/command on the gateway's listeners over the Docker network; the public proxy does not forward it. The command's token, checked in constant time before anything else, is the route's only authentication, and there is no replay protection; bodies over 64 KiB are refused. The answer is an ephemeral acknowledgement and the result is posted by the bot, in the thread when the callback carries root_id. The other way in is a message to the bot: @kapelle status or a direct message.
• Identity and rooms: a person is a Mattermost user id in the directory (user_identities); /me link mattermost <user id> is confirmed by sending the code to the bot. A channel that is linked to no team is ignored except for /team and /help.

Linear¶

• Install: OAuth with actor=app, scopes app:assignable and app:mentionable, with the agent session events webhook enabled.
• Starting work: delegating or mentioning creates an AgentSession and sends an AgentSessionEvent (created, prompted). Respond within 5 seconds and post a thought within 10. Waking an agent can take longer, so the gateway posts the first thought itself.
• Activities:
• thought and action: activity feed
• elicitation: INPUT_REQUIRED
• response: COMPLETED
• error: FAILED
• the stop signal: CancelTask

Jira¶

• Preferred: Forge remote agent (preview). rovo:agentConnector with protocols.agent2Agent.version: "1.0". Jira calls the lead's A2A HTTP interface with SendMessage, GetTask, SubscribeToTask and CancelTask. The gateway verifies the Forge Invocation Token against its JWKS keys.
• Fallback: OAuth 2.0 (3LO). POST /rest/api/3/webhook registers webhooks, which expire after 30 days and are refreshed with PUT /rest/api/3/webhook/refresh. Comments use ADF; status changes use transitions.
• Avoid Connect: end of support begins 2027-01-31.

GitHub Issues¶

• Auth: a GitHub App whose installation tokens last 1 hour, scoped to one repo and specific permissions. Agents never hold them; the credential gateway adds them to git and API requests (§9).
• Triggers: the label agent:<team> (issues.labeled) starts work, and replies come from issue_comment.created. Issues can't be assigned to Apps.
• Results: a PR with Closes #N, which links only when it targets the default branch, plus check runs for status.

Commands and replies, on every surface¶

On Mattermost the leading / is written /kapelle (the slash command) or @kapelle (a mention).

On Slack, /status and /cancel are typed as a plain reply inside the work item's own thread, not as a registered slash command: a Slack slash command's payload carries no thread information and cannot be invoked from inside a thread at all, so a command that needs a specific work item cannot be one there. /budget needs no thread (it answers for the team, not one work item), so it is a registered slash command, same as /team; /status is additionally Slack's own built-in command name (sets a person's away/active status), a second reason never to register ours under that name (backlog task 85db7a3f).

People and permissions¶

Surface identities (a Slack user id, a Mattermost account, a GitHub login, a Linear or Atlassian account) are linked to one person in a user directory in Postgres, with roles per team. Nothing on any surface is acted on unless it comes from a linked person with the right role.

• Public repositories. On GitHub only the label from a team member starts work, and only comments from linked people become messages. Everything else on the issue is visible to the agent as content, never as instruction (§17).
• Linking identities. A person links a new surface identity from the home room (/me link github <login>), confirmed from the other side, so no one can claim someone else's account.
• Feedback while waiting. When a task is queued behind busy replicas or its agent is waking, the adapter posts "queued" or "waking" in the thread or issue, so silence never means lost. Linear's 10-second thought deadline is met the same way.
• Idempotency. GitHub and Linear redeliver webhooks; each delivery id is recorded and duplicates are dropped.
• Verification. Signed webhooks (GitHub HMAC, Jira, Linear) and Forge tokens are verified at the gateway before anything else happens. An optional second listener (KAPELLE_GATEWAY_WEBHOOK_LISTEN) serves only the webhook routes and a health path, for a public reverse proxy; the A2A routes are never on it.

Team memory¶

Each team has a shared memory: the conventions, decisions, gotchas and past outcomes that don't live in any one repo or conversation. Every role and replica reads the same memory, so the second work item on a subject benefits from the first, and a coder started today knows what the reviewer insisted on last month. Memory is a service in the control plane, reached through the worker's MCP tools like everything else; nothing is copied onto agent disks.

What it stores¶

An entry has: team, kind, title, body (Markdown, at most 2 KB), tags, source (person, agent role, task), status (proposed or curated), created and updated times, and an optional supersedes link. Entries are never edited in place by agents; a correction is a new entry that supersedes the old one, so history stays readable.

How agents use it¶

• At the start of a run. The executor searches memory with the task text, the role and the repo, and prepends a "Team memory" block to the first message: curated entries first, then proposed ones, capped at about 1,500 tokens. The block is labelled as notes from the team, not instructions.
• During a run. MCP tools on the worker (§7): memory_search(query, kind?), memory_get(id), memory_propose(kind, title, body, tags). Proposals from agents are always proposed.
• At the end. When a work item completes, the executor writes an outcome entry from the artifacts and the lead's final message, linked to the work item and PR. Outcomes are the memory of what the team has done.
• Between roles. Delegation messages stay as they are; memory is for what should outlive a work item, not a side channel for the current one.

Curation and safety¶

• People curate. Every agent proposal is posted in the home room. /memory accept <id>, /memory edit <id>, /memory forget <id>, /memory add <kind> …, /memory list and /memory search … work on every surface's home room. Accepted entries become curated.
• Proposals expire. A proposed entry that nobody accepts is dropped after 30 days, so memory can't silently fill with an agent's guesses.
• Memory is data. Agents write memory after reading untrusted content, so an injected instruction could try to persist itself. Role prompts treat memory entries as notes, never as commands; proposals are shown to people before they carry weight; entries are scanned for credential patterns and URLs to unknown hosts before being stored.
• Team-scoped. The agent token only reaches its own team's memory. People read and write memory for teams they are members of (§13).
• Audited. Reads that were injected into a run and every write appear in the activity feed and traces (§16).

Storage¶

• Postgres with pgvector, in the existing kapelle database: an entries table partitioned by team, a tsvector column for keyword search and an embedding column for semantic search; queries combine both. No new stateful component.
• Embeddings through LiteLLM under an embed alias, so the embedding model is as replaceable as the chat models (§10). The Aiven AI Gateway serves no embedding model as of 2026-09-24, so no embed alias is configured and KAPELLE_EMBED_ALIAS stays unset: search is keyword-only until the gateway offers one, never a hard dependency.
• A MemoryStore interface in the controller (search, get, propose, accept, supersede, forget, export) with the Postgres implementation first, so a hosted memory service could replace it later without touching workers.

Relationship to the repository¶

Conventions that belong with the code stay in the repo (AGENTS.md, CONTRIBUTING.md), which every role reads from its worktree. Team memory holds what a repo can't: decisions across repos, people's preferences, operational gotchas and the history of work items. /memory export writes the curated entries into docs/team-memory.md in the team's repo as a PR, so the two never drift far apart and the repo remains the record if the platform is ever retired.

Skills and MCP servers¶

Memory (§14) records what a team has learned; a skill packages how to do something so any role can do it the same way tomorrow: how this team releases a service, how to run the flaky integration suite, how to write a migration for the billing schema, how to review a Terraform change. A skill is a directory with a SKILL.md and, optionally, scripts, references and assets, in the AgentSkills format that the OpenHands SDK already loads (openhands.sdk.skills) and that Claude Code and other tools share, so a skill written for Kapelle is portable and a skill written elsewhere can be dropped in.

What a skill is¶

release-service/
  SKILL.md          # frontmatter: name, description, triggers, allowed_tools, version
  scripts/bump.py   # runnable helpers, invoked by the agent through its normal tools
  references/       # longer material the agent reads on demand

SKILL.md frontmatter carries name, description, version, optional triggers (keywords or globs that activate the skill when a task mentions them), allowed_tools and compatibility. The body is the procedure. A skill without triggers is listed to the agent by name and description only and read when the agent decides it applies; a skill with triggers is also injected when a task matches. The description is the contract: the agent chooses skills by it, so it says when to use the skill, not what the skill is about.

Where skills come from¶

Skills merge by name in that order, later sources overriding earlier ones, so a repository can specialise a platform skill and a team can pin its own version without forking the template. A role's definition can exclude skills that don't apply to it (a planner never needs the release procedure). A skill named by a role's own definition or a team's template is required: if it can't be resolved, or resolves but doesn't fully land in the guest, the run fails rather than continuing without it — a run that pretends to have a skill is worse than a run that fails. Repository and team skills stay best-effort: one that fails to resolve or land is left out, with a warning, and the run continues.

How agents use them¶

• At run start. The worker resolves the four sources for the team, role and repository and hands the merged set to the runtime. For OpenHands that is the SDK's own skills loader on the AgentContext, so listing, triggering and file access behave exactly as upstream documents. For the Pydantic AI and Deep Agents runtimes the same set is rendered as an "Available skills" section in the system prompt, with SKILL.md and its resources readable through the agent's file tools; the two behaviours are covered by the same conformance test.
• During a run. A triggered skill's body is injected once; scripts run in the sandbox through the agent's usual shell tool, so they are subject to the same credential gateway and egress policy as anything else the agent runs (§9). A skill never carries secrets; it names the destination and the gateway supplies the credential.
• Feedback into memory. When a run that used a skill ends, the outcome entry (§14) records which skills were active, so a reviewer can see that "the release went wrong" and "the release skill changed last week" are the same story.

Curation and safety¶

• People own skills. Agents propose a skill the way they propose memory: skill_propose(name, body) from the worker's MCP tools posts a draft in the home room; /skills accept <name> stores it as a team skill, /skills list, /skills show <name>, /skills forget <name> manage the set. Repository and platform skills change only through pull requests.
• Skills are instructions, so they are reviewed like code. A skill can tell an agent to run a script; an injected skill is the most direct prompt-injection vector this design has. Repository skills are loaded only from the default branch the team was created against, never from an agent's own working branch, and every skill body passes the same credential and unknown-host scan as memory entries before it is stored or injected.
• Versioned and attributed. Each skill carries a version and the run's activity feed (§16) records the name and version of every skill injected, so an outcome can be traced to the procedure that produced it.
• Budgeted. The listing is names and descriptions only; a body is injected on trigger or on request, capped per run the way the memory block is, so a large skill library costs tokens only when used.

MCP servers¶

Skills give an agent procedures; MCP servers give it tools it doesn't have: a ticket tracker, a documentation index, a database console, a browser. Today the only MCP server an agent sees is the worker's own (§7: delegation and memory tools). External servers follow the same sourcing and curation as skills, and their credentials follow the same rule as every other secret in this design: they never enter the VM.

Declaring a server. An MCP server is declared in the same four places a skill can come from (platform roles/mcp/<name>.yaml, a template's mcp_servers: list, a repository's .kapelle/mcp/<name>.yaml, or a team's stored set through /mcp commands), merged by name in the same order. A skill may also bundle the servers it needs (the SDK's Skill.mcp_tools), so "how to triage a Linear ticket" ships with the Linear server it uses. A declaration is the SDK's MCPServer shape plus a destination field:

name: linear
transport: http                      # http (streamable HTTP) or stdio
url: https://mcp.linear.app/mcp
destination: linear-mcp              # the credential gateway destination that carries the secret (§9)
roles: [planner, reviewer]           # optional; default every role
tools: {allow: ["*"], deny: ["delete_*"]}

How credentials flow. The agent's MCP client is configured exactly as the LLM client is: the literal placeholder as the bearer token, API key or basic password (§9, placeholder-header contract). Every declared server is a named destination in the role's credential-gateway policy (host, allowed paths, header_allowlist that includes the MCP routing headers, inject_secret), and the real secret lives in OpenBao under the team, minted or stored by the controller the way GitHub installation tokens are. iron-proxy swaps the placeholder on the way out, so a server declared without a matching destination is unreachable rather than insecure. Servers that need no credential get a pass-through destination, still allowlisted by host and path.

• Remote servers (streamable HTTP) are the default: the guest connects through the proxy, the proxy injects the credential, the audit log records every call. OAuth-protected servers get their token minted and refreshed by the controller and served through the same injection; the VM never sees a refresh token.
• Local servers (stdio) run inside the sandbox when a server has to be next to the code (a language server, a test runner). They are started with placeholder in their environment, so any HTTP call they make to an allowed destination gets the real credential from the proxy exactly like the agent's own tools; a stdio server whose upstream is not a declared destination simply can't reach it. A stdio server never receives a real secret through its environment or arguments.
• Per-role tool filters. roles and tools.allow/deny are applied by the worker when it hands the server list to the runtime (the SDK's tool filters for OpenHands, the adapter's for the other runtimes), so a reviewer can read tickets through a server the planner may also write to.

Curation and safety. The same as for skills: people accept a proposed server before it applies; repository declarations load only from the default branch; the server list, the tools actually called and the destination each call left through appear in the activity feed and traces (§16). Tool results are untrusted input like any other content an agent reads. A declaration cannot widen a role's credential policy: the destination has to exist in the role's policy file already, so adding a server to a repository never grants a credential by itself.

Delegation of identity. Everything above uses a team credential. Acting as a specific person on a tracker (so a ticket comment appears under their name) needs per-person tokens; that is the token-exchange case the credentials section keeps for later (agentgateway, §9, "Implementation options"), and it slots in as another inject_secret source without changing the agent side.

Observability¶

• Team room. Each work item's Mattermost thread gets every message between roles, status changes, artifacts, batched activity summaries, a trace link and the cost so far.
• Activity feed. Executor callbacks receive OpenHands events and publish summaries to the activity subject.
• Traces. OTEL_EXPORTER_OTLP_ENDPOINT is set on agent-servers (OpenHands tracing is built on Laminar but needs no Laminar account); workers, gateway and controller read KAPELLE_OTEL_TRACES_ENDPOINT instead (the standard name would start the SDK's gRPC Laminar exporter in the worker). Trace context travels in NATS headers.
• Agent VMs. vmd exports metrics per agent and per host:
• state (running, idle, asleep, deep sleep)
• wake time, split into snapshot restore and cold boot
• snapshot size
• disk usage and memory
• watchdog kills
• Service metrics. The worker (kapelle_worker.metrics) and the gateway (kapelle_gateway.metrics) each export Prometheus metrics on their own standalone /metrics ASGI server (own CollectorRegistry, own uvicorn.Server + port -- KAPELLE_METRICS_HOST/KAPELLE_METRICS_PORT, matching vmd/credgw's own -metrics-listen pattern), independently of whichever other HTTP surfaces each process happens to have running.
• Spend. LiteLLM spend per agent key and per team feeds the cost line.
• Credential audit. The credential gateway logs every request it adds a credential to (agent, task, destination, method, path, status) and exports it to the collector.
• Telemetry off. DO_NOT_TRACK=1 on agent-servers, with no Laminar project keys.

Operations and security¶

• Credentials. VMs hold no outbound credentials; the credential gateway adds them (§9).
• Provider API keys exist only in LiteLLM.
• LiteLLM keys, the GitHub App key and surface credentials live in OpenBao.
• Session keys and OH_SECRET_KEY reach VMs through MMDS.
• Assume an agent VM is compromised. It can't read credentials, but it can use them through the gateway within its policy. Narrow host, path and repo scopes, budgets, the audit log and per-team agents limit the damage.
• Untrusted input. Everything an agent reads can carry instructions: issue text, comments, PR reviews, repository contents, tool output. Credentials being outside the VM limits what an injected instruction can steal, so the remaining controls are about what it can change:
• Agents never merge. Every agent PR needs a human review, and agent-authored PRs are labelled as such.
• The GitHub App's token can push only to agent/<team>/*, enforced by repository rulesets (to be verified); the gateway refuses pushes to any other ref as a second check.
• Role prompts mark surface text and tool output as data, not instructions, and the evaluation suite includes injection cases.
• Only messages from linked people become instructions (§13); an agent sees other comments as content.
• Snapshot and disk hygiene. Memory snapshots are encrypted at rest and deleted after one restore. Data disks are encrypted, backed up nightly while sleeping and on deep sleep, and restored on host loss (§8). Hosts have no shared storage.
• Host fleet.
• Hosts: KVM hosts with cgroups v2 and a supported host kernel (5.10, 6.1 or 6.18) that stays the same while agents sleep on them.
• Upgrades: drain a host by moving its agents to deep sleep first.
• Snapshot compatibility: all hosts in a pool use the same CPU model or a custom CPU template.
• Inbound traffic. Slack connects outbound over Socket Mode. Linear, GitHub and Jira reach a public HTTPS endpoint on the gateway. Agent-servers are reachable only from worker hosts.
• Upgrades. Pin openhands-sdk and the VM image, Firecracker, guest kernel, LiteLLM and vLLM, and upgrade one at a time behind the evaluation suite. A Firecracker or kernel upgrade invalidates snapshots, so agents cold-boot afterwards. An OpenHands upgrade is done as a deep-sleep cycle: in-flight work items finish on the old image, conversations are archived, and the agent boots the new image with empty conversation state, because persisted conversation events are not guaranteed to load across versions.

Runtime evaluation¶

OpenHands was chosen from the open-source agent SDKs checked on 2026-09-12. None supports A2A over a broker, so the deciding factors were coding tools that work with self-hosted models, sandbox support, and how much each project pushes toward its own paid services.

Evaluation suite¶

1. Pick 5 to 10 real tasks from our repos (bug fix, feature, refactor, tests, review), each with an automatic pass/fail check.
2. Run each with OpenHands, the Pydantic AI harness and Deep Agents, in the same microVM profile and with the same LiteLLM alias, three runs each.
3. Record pass rate, tokens and cost, wall time, iterations and failure types.
4. Repeat for each candidate model alias, and admit models per role above a pass-rate threshold.
5. Also measure agent wake times (snapshot restore and cold boot) and memory per agent.
6. Keep the suite as the gate for runtime, image, VMM and model upgrades.

Decisions¶

Risks and open questions¶

• vmd is ours. Snapshot handling, networking, jailer setup and encryption are security-critical code we own. Mitigated by reusing E2B Runtime patterns, keeping vmd small, and reviewing it separately.
• Snapshot limits. Snapshots are tied to one host, Firecracker version, kernel and CPU. A host loss or upgrade means cold boots. Memory files need disk equal to guest RAM per sleeping agent.
• After restore. Clock resync, dropped connections, rewriting MMDS and random-number state all need tests. Firecracker's random-for-clones guidance (VMGenID, deleting random-seed) applies even though we never clone.
• Wake times are unmeasured. The only official figures are a 125 ms boot-to-init target and an anecdotal 3 ms restore. Agent-server start-up and reconnect time need measuring on real hosts.
• The credential gateway is a high-value target. It holds live short-lived tokens for every agent on its host. Mitigated by keeping long-lived keys in OpenBao only, a small code base, access restricted to its host's VMs, and monitoring.
• TLS interception compatibility. Tools that pin certificates, ship their own CA bundle or ignore proxy settings will fail. Each role's toolchain has to be run through the evaluation suite behind the gateway.
• Misuse through the gateway. An agent can't read credentials but can use them within its policy. Branch-level push limits (GitHub rulesets, and the gateway inspecting push refs) are load-bearing for the prompt-injection controls in §17 and haven't been checked.
• Prompt injection. The controls in §17 limit damage to a reviewable PR on an agent branch; they don't stop an agent from wasting its budget or producing misleading output. Human review of every agent PR is the real guard, and it needs to stay a policy people follow.
• Question detection. The ask_requester fallback (a run ending in a question with no artifact) is a heuristic and will misclassify some runs in both directions. The evaluation suite measures it; the team room shows both outcomes so a person can correct them.
• Memory quality. Injected memory competes with the task for context, and stale or wrong entries mislead every later run. Curation, expiry of unaccepted proposals, the token cap on the injected block and outcome entries linked to their PRs keep it checkable; the evaluation suite should include runs with and without memory to show it helps.
• Gateway building blocks. Whether iron-proxy can identify agents by network source hasn't been checked. The OpenBao GitHub token plugin is a small fork, so the controller minting tokens itself is the fallback.
• Shrinking sleeping agents. Whether the balloon or virtio-mem unplugging before a snapshot reduces the memory file needs a test.
• Local development. Linux with KVM only. macOS works only on M3 or later via Lima nested virtualization (Firecracker untested there). vmd needs privileges for taps and nftables on laptops.
• Cloud overhead. Nested virtualization costs at least 10% CPU on GCP. Firecracker itself is only tested on metal.
• Not yet checked in OpenHands: how MCP servers are configured on an Agent (needed for ask_requester), whether a standalone bash command can still run while all conversations are idle, and the agent-server's real memory footprint.
• Previews. Jira Forge remote agents and some Slack agent session features are previews or need paid plans.
• Non-standard transport. Only our components speak A2A over NATS. The adapter stays isolated so an official binding could replace it.

Sources¶

• A2A Protocol Specification
• a2a-python SDK
• OpenHands Software Agent SDK
• OpenHands docs
• Firecracker snapshot support
• Firecracker snapshot versioning
• Firecracker random for clones
• Firecracker production host setup
• Firecracker jailer
• Firecracker network setup
• Firecracker rootfs and kernel setup
• Firecracker kernel policy
• Firecracker MMDS
• Firecracker balloon
• E2B Runtime
• Flintlock
• cocoonstack/sandbox
• crucible
• Cloud Hypervisor
• AWS EC2 nested virtualization
• GCP nested virtualization
• Lima nested virtualization option
• LiteLLM virtual keys
• LiteLLM enterprise features
• Slack Socket Mode
• Mattermost API reference
• Linear agents
• Jira remote agents (Forge)
• GitHub App installation tokens
• OpenBao
• pgvector
• iron-proxy
• Hermes Agent: iron-proxy egress
• Envoy credential injector filter
• Infisical Agent Vault
• agentgateway: credential injection patterns
• Pipelock
• GitHub: authenticating as an app installation
• actions/checkout git auth helper
• openbao-plugin-secrets-github