How to read this guide
This guide is for the person who decides how a system on AgentSpaces is shaped: how many peers, where they run, which boundaries the application draws, what each space promises, and where the fleet meets the rest of the enterprise. It assumes some working knowledge of distributed systems. It's a companion of the developer guide for the API and the workflow guide for the stage-by-stage patterns; this guide decides what those two build.
We designed AgentSpaces to bookend a story that began at Bell Labs in 1986 with the publication of Linda, continued on through the partnership with Sun Microsystems and their release of Jini and JXTA. When we arrived at Lucent in 96 and '00, we reviewed, designed, and prototyped with Jini and JXTA, though didn't wind up including it on any of the product lines we architected. We're ultimately coming full circle as the Bell Labs crew comes back together again at Bad Monkey to combine the best parts of Jini and JXTA with a healthy dose of modernization on related standards (PKI, TLS, mTLS, DTLS, CBOR, etc). While P2P found a home in CDNs and other large-scale economic saving endeavors, technical improvements have been driven by a steady state of research in the intervening years. If we implement one of those papers, we'll cite it in the documentation.
The system in one picture
A peer is one process running the library. It has an Ed25519 identity, joins one or more groups, and registers the spaces it takes part in. A group is the set of peers that gossip with one another: membership by SWIM probes, advertisements by a gossip stream, space state by anti-entropy. A space is a replicated tuple space: typed entries with leases, written, read, taken under a lease, and completed, all as signed records folded into a CRDT so that every replica converges whatever order the records arrive in. Capabilities (votes, push-sum aggregates, the ordered log, semantic discovery, gossip learning) run on pipes between peers of the group. Agents are ordinary objects bound to spaces and capabilities by annotations, or by the equivalent decorators in Python and TypeScript and the defagent form in Clojure; each publishes a signed card the others discover.
Four consequences shape every architecture on it. There is no primary and no broker, so there is nothing to fail over and nothing to scale separately; a peer that registers a space holds all of it. Convergence has a rate, set by one tick clock per peer (250 ms by default) and a gossip period (1 s) carried in the group's advertisement: a write reaches every replica in a few periods, and the settle windows, leases, and deadlines of chapters 6 to 9 are multiples of that rate. A guarantee stronger than replication (one taker per entry, one completion fleet-wide, a decision) is a protocol run on demand on the spaces that need it, so each space stands on its own rung of the ladder in chapter 7. And identity is on the wire: each record carries its issuer's signature, so the trust perimeter is membership and admission rather than a network edge.
SPEC.md and TECH-SPEC.md (section 2 of the technical specification, "Why these protocols", is the rationale this guide builds on).How many peers, and what runs where
Count peers by what they hold; the number of agents is a separate question. An agent is a binding on a peer, and a peer can host as many agents as its process can run; three peers can seat thirty agents. A peer is a unit of replication (a full copy of each space it registers), a unit of failure (its take leases lapse together), a unit of trust (one identity, one keystore, one channel), and, for the ordered log, a unit of quorum. Decide the peer count on those four axes and let agent placement follow.
Three sizes
| Size | What it looks like | What changes | Where it is shown |
|---|---|---|---|
| 3 to 5 peers | one site, every peer listens, every member is a seed | nothing special: full views, full replicas, one ordered log needs a majority of 2 of 3 or 3 of 5 | examples 02, 12, 18 (three peers, five spaces); the Spring, Micronaut, and LangChain4j example fleets |
| 5 to 15 peers | a site or two, a rendezvous, roles emerge (console, gateway, relay) | advertisement caches fill (8,192 entries by default, 32,768 on a rendezvous); keep card TTLs and refresh at the defaults; decide which few peers hold the ordered log's member set | the Party Bus: 15 peers, 9 spaces, 34 agents on TLS |
| Beyond that | several sites, many dial-only workers, partner groups | replication is full by default, so put a large space on a tag-sharded replica only on the peers that need it (builder-only today); split by group rather than by space when the perimeter differs; expect no published scale ceiling, measure with the stress harnesses in agentspaces-perf and the traffic audits | a fifty-member simulated fleet in the peering suite (RobustnessScaleTest) idles at about five frames per member per tick and drops a crashed member from every view within its TTL; six real peers idle at 0.7 frames per peer per second, the same across two hosts (examples/superfriends/two-host-audit.sh, PERF2 ยง5) |
The library publishes no peer-count limit. Three things grow with the fleet: membership views (full; every peer knows every member), replication (full per registered space; every replica holds every entry until its tombstone collects), and the advertisement cache (bounded, evicting on read). Gossip propagation is O(log N) rounds, and a per-issuer token bucket (50,000 frames, 10,000 a second) bounds what any one peer can push. The one cost that is linear per member is the advertisement refresh, which goes directly to every member: once a fleet outgrows what the probe cycle can keep alive inside the member TTL, the refresh runs at half the TTL and is the keepalive, so an idle member costs about (N - 1) frames per refresh interval; probes and anti-entropy back off while nothing moves. None of these is a cliff at ten or fifty peers, but each is a reason to keep a group to one purpose and to use several groups for several purposes.
Roles
Two roles exist on the wire, and three more are conventions you adopt. RENDEZVOUS is a peer that keeps a large advertisement cache and answers discovery queries directly; a site seeds from it. RELAY is a peer that forwards a fully signed frame from one member to another, one hop only, so that two dial-only peers can reach each other; relays never re-relay. By convention a console peer is a read-only member that serves the operator's view and, with the directive grant, command-and-control; a gateway peer fronts the fleet for A2A or MCP clients; a coordinator peer holds an ordered log's leadership most of the time, though any member of the log's set may lead.
| Decision | Rule | Why |
|---|---|---|
| Workers per kind | three or four, on different peers | one-of-N availability stops improving past that point and the floor (shared provider, shared seed) takes over; see chapter 6 |
| Ordered logs | one per group per peer, a fixed member set of three or five | a quorum is a majority; reconfiguration means a new log; keep the set on the peers with the steadiest uptime |
| Rendezvous | one per site, two where the site must not lose discovery | it is a cache, never an authority; losing it slows discovery and changes no result |
| Console and gateway | their own peers, read-only or edge-facing | they hold no take leases, so their restarts cost the flow nothing, and their inbound ports are the only ones you expose |
| Dial-only peers | as many as you like | a peer with no bind advertises no endpoint, speaks only to what it dialed, and needs a relay to speak to another dial-only peer |
Discovery and bootstrap
The options
A peer joins a group in one of two ways. With a founding string, every member derives the same group id locally from the same document, so any member is a seed and nobody holds a secret; this is what the starter's founding: does and what every example uses. With join by group id, the peer dials a seed, asks for the group's founding document, verifies that the document hashes to exactly the id it wanted, and only then introduces itself; a hostile seed can delay that join and can never substitute a policy. The first is right inside one organization; the second is right when the id travels further than the document (a partner's configuration, a printed credential, a registry).
| Mechanism | What it gives | What it costs | When |
|---|---|---|---|
Seeds (groups[].seeds) | a way in; a member once joined needs them no more | a list to maintain; a seed that is down is only a delay if there is another | always; give every peer at least two |
| Rendezvous role | a large advertisement cache that answers queries directly and seeds a whole site | one more peer to run; none to correctness | a site, a WAN, any fleet past a handful of peers |
| Relay role | one signed hop between two peers that cannot dial each other | a relay peer per region; relayed frames are refused under required attestation | dial-only peers that must reach each other; NAT on both ends |
Multicast beacon (agentspaces.multicast.*, off by default) | LAN peers find a seed without configuration | UDP on the segment; only a hint under required attestation | a lab, a vehicle, a floor of devices on one segment |
| Join by group id | a verified founding document before the first frame | a reachable seed that joined with the signed document; a timeout (10 s by default) | ids that cross an organizational line |
There is no universal bootstrap group and no mDNS; a deployment that needs a directory of groups founds one and seeds from it. The rendezvous and relay roles are declared by the peer that plays them (agentspaces.roles) and learned by everyone else from its advertisement.
Cards and advertisements
Every discoverable thing is a signed, typed, time-limited document: a peer advertisement (TTL 10 minutes), an agent card, a space advertisement, a capability advertisement (15 minutes each), the group advertisement (24 hours). They travel on one gossip stream, every cache keeps one per id and replaces it only with a newer issue, and a refresh cadence (card-refresh-millis, 5 minutes by default) keeps live things live. Revocations travel the same way and are never cached: what a replica already holds stays, and what arrives after is refused. Semantic discovery ranks cards by an embedding (a deterministic hashing embedder by default, a real model when you give one) and asks up to three peers over a pipe for theirs; it is a routing hint, never a roster.
The design rule that follows: route work by entry type and treat a discovered name as a hint. An agent card tells a planner what is available and a remote caller how to address it; the space and the record type are what actually deliver the work, and they do so whether or not the card has arrived yet. A fleet whose correctness depends on a discovery result has put an eventually consistent cache on its critical path.
OPEN, uses the default gossip parameters and SWIM configuration, and is founded by a generated key the starter does not keep. Operations the specification reserves to the founder, revoking a peer and rotating the content key without a grant, therefore need either a group founded in code with a kept founder key (PeerNode.joinGroup(SignedGroupAdvertisement, ...)) or the grants the starter does expose (security.grants.key-rotator, the founder-or-ca revocation validator). Decide which before the fleet has data in it. The membership policies INVITE and POLICY are likewise hand-wired today. See the developer guide's security chapter.Mixing peers: hosts, languages, profiles
One group may hold peers on every host the project supports: the Spring Boot starter, Micronaut, LangChain4j on a plain JVM, Embabel, a peer built from the core API, and the Python, TypeScript, and Clojure clients. They share one wire (CBOR, Ed25519, wire version 3, pinned by golden vectors every client runs), so the question is never whether they can talk but what each can host. Decide placement by the table, then write each agent in the language its team writes.
| Peer kind | Hosts | Does not host | Posture |
|---|---|---|---|
| JVM host (Spring, Micronaut, LangChain4j, Embabel, core) | every space strategy, every capability including the ordered log, A2A and MCP gateways, the console, model serving | listens or dials; TLS by default (mtls) | |
Clojure (agentspaces-clj) | the same as a JVM host through fleet/start and defagent, including ordered takes and joins | the same | |
| Python client | takes, notifies, bids, ballots, decisions, proposals, estimates, LOCAL joins, push-sum, semantic queries, remote actions, its own agent certificate | the ordered log, so no ordered takes, reduces, or fleet-wide join modes; gossip learning | dial-only on plain TCP; sends capability frames only to members it can reach |
| TypeScript client | the same as Python | the same | the same |
Three rules keep a mixed fleet coherent. Profiles must meet. Note that in 0.3.0, a non-JVM client speaks plain TCP, so the peer it dials must listen on TCP: either the fleet runs dev-local (signed frames, no channel encryption) or a JVM peer fronts the clients with a TCP listener while the JVM core stays on TLS; a mixed fleet degrades to its lowest rung unless require-attestation refuses the plaintext link outright. We expect 1.0.0 to provide the same transport options universally across all libraries, we started with Java so it proves the architecture and then we uplift python, typescript, etc.Once-only work lives on a JVM peer. A Python or TypeScript worker takes under the lease race and completes atomically like any other, but a stage that needs the ordered log binds on a JVM host, and the clients write into and read from that space. Capabilities announce themselves. A peer that runs an aggregate advertises it, and shares go only to members whose advertisement is live, so a bare client or a console never swallows mass; a client that wants to contribute starts its aggregate and advertises like everyone else.
CouncilFleet in example 02 runs a Java coordinator with Python and TypeScript council workers in their own processes: tasks taken, a vote decided at a quorum of three, a push-sum epoch settled at the fleet mean, a semantic query answered over the pipe, a remote action served. run-trilingual-demo.sh plays it as its second act. The developer guide's language chapter has each client's surface.Spaces as lanes, groups as perimeters
One space per lane
A space carries one strategy (LEASE_RACE or AUCTION; the ordered log is a coordinator registered on a space rather than a third strategy), one admission rule, one replication mode, one settle window, and one retention policy through its leases. So the unit of design is the lane: give each stage of the application its own space the way you would give it its own queue, and let the record types say what flows. Example 18 runs a nine-stage claims flow over five spaces (intake, assessment, decisions, payments, ledger), each with the strategy its stage needs: the assessment space is an auction because adjusters price work, the payments space hosts the ordered log because a payment happens once, the ledger holds reduce accumulators because exposure is a running total.
| Draw a new space when | Keep one space when |
|---|---|
| the strategy differs (one lane races, another auctions) | the records are stages of one case and the same workers read them |
| admission differs (a lane only an attested agent may write) | a reader wants one template over the whole lane |
| retention differs (hours for tasks, days for an audit trail) | a join draws parts from them (a part may name another space; the join's own space holds its tickets) |
| the ordered log must live there (one log per peer per group) | the volume is small enough that a full replica everywhere costs nothing |
Within a space, tags and filters partition without a new space: a template matches on tags before it decodes a payload, so a keyless relay can route by tag, and a where filter narrows a worker to the entries addressed to it. Example 18 deploys one shop class per region on one space. Reach for a tag shard (TAG_SHARDED replication, builder-only today) when a space is large and most peers need one slice of it.
A group per tenant, mission, or conversation
A group is the membership perimeter: who gossips, who sees the advertisements, whose signatures count. One peer may belong to several, and an agent binds into a named group (group = "..." on every annotation, or every satisfying group when unnamed) with one card published per group. Draw a new group when the set of parties differs: a tenant, a mission with its own lifetime, a partner organization, a conversation that should leave no trace in the others. Draw a new space when only the lane differs. The bridge between two groups is a peer that is a member of both and hosts an agent in each; it hands records across by taking in one and writing in the other, and it is the one place a cross-group policy can be enforced.
| Boundary | Mechanism | Starter support |
|---|---|---|
| Who may join | membership policy OPEN, INVITE (a founder-signed credential), POLICY (a validator SPI, for instance DID) | OPEN; the other two are wired in code |
| Who may write or take a space | space admission GROUP, ALLOWLIST, CREDENTIAL, AUTHORIZER, judged per acting agent; reads stay open to the group | spaces[].admission, allowed-agents, credential-issuer |
| Who may read a payload | a group content key (AES-256-GCM); members without it relay and cannot read; rotation with a cutover delay | groups[].content-key and content-key-rotation |
| Who may do an operation | the authorizer's nine operations under security.grants.*, or OIDC scopes | yes |
Sharing across teams and organizations
Sharing inside one organization is a question of lanes and admission; sharing across organizations is a question of trust roots. The fabric gives you four independent levers, and the architecture is which of them you pull for which boundary.
| Lever | What it decides | Granularity | Where |
|---|---|---|---|
| Membership | who is in the group at all | peer | the founding document's policy; INVITE credentials; a POLICY validator |
| Admission | who may write or take a given space | acting agent | the space advertisement; allowlists and credentials |
| Authorization | who may vote, take, write, serve a model, rotate a key, issue a directive | peer, or peer/agent with attested agents | MembershipAuthorizer grants, or an OidcAuthorizer with aspace:<op> scopes |
| Confidentiality | who may read a payload | key holder | group content keys, served by key-wrap, rotated on a schedule |
Two teams in one organization usually share a group and separate by space and admission: team A's registrars write intake, team B's adjusters take assessment, both read decisions, and the authorizer's VOTE grant names the panel. Two organizations usually get two groups and a bridge: each keeps its own founder, keystore, and content key; the bridge peer is a member of both, admitted on each side under that side's policy, and the records it carries across are the contract between them. Where a partner should see one lane and nothing else, give that lane its own group rather than its own space, since reads in a space are open to the whole group.
Identity is what makes the levers compose. Every record carries its issuer's signature, so a replica can judge admission and grants without asking anyone. An agent can act under its own certified key (agent-attested) rather than its peer's, which is what lets a grant name one agent on a shared peer and what the AGENT vote granularity counts. Revocation is eventually consistent and follows a freeze rule: what a replica holds stays, what arrives after the revocation is refused, and a key-compromise revocation refuses everything new at once. Plan the revocation path before the first partner joins: who issues it (the founder, or a CA under founder-or-ca), how it reaches every replica (gossip, within a few periods), and what the frozen history means for the records already shared.
POLICY group with DID credentials for membership, a group content key with key-wrap so a relay can carry what it cannot read, an ORDERED space for the commitments both sides must see once, and an A2A gateway for partners that run no peer at all. The use-case models on the site sketch it; the pieces each have a chapter in the developer guide.Building up reliability
Reliability on this fabric is built from four mechanisms you compose rather than one you switch on. The lease is recovery: a take holds an entry for a declared time, and a worker that dies lets it lapse, after which the entry is available to any sibling; nothing schedules a retry and nothing has to notice the death. The count is availability: bind the same worker on several peers and the lane drains while any one lives. Replication is durability within the group: an entry lives on every replica from its first gossip period, and a crashed peer's replica is rebuilt by anti-entropy when it returns. And the ordered log is the one place a majority is required, so it is the one place a partition costs availability.
| Question | Answer | Knob |
|---|---|---|
| How long until a crashed worker's task is retried? | the take lease, plus one tick | lease on @SpaceTake: 30 s for a quick task, 2 m for a model call, 10 m for a long job; always longer than the work |
| How long until a crashed peer is forgotten? | the member TTL (30 s by default) | SWIM configuration, in code |
| What if a link drops, or two members never met? | a JVM peer retries a lost member's last endpoints and re-dials its seeds when its view is empty, backing off to thirty seconds; the view itself is anti-entropy state, so a member whose introduction was lost is learned from any partner; a Python or TypeScript peer re-dials its seed, re-introduces itself, and re-pulls what it had | nothing to configure; ROBUSTNESS.md has the scenarios |
| What happens during a partition? | each side keeps writing to its replicas and they merge on heal; a lease race may complete the same entry on both sides once; the ordered log's minority side cannot take | which spaces stand on the ordered log (chapter 7) |
| What does a reaction do on failure? | nothing: a @SpaceNotify that throws is logged and dropped | put work that must happen behind a take; put nice-to-have reactions behind a notify |
| How do I see it? | the console: per space the written, completed, queued, and in-progress counts; per peer the suspect state and the channel mode; the wire-traffic panel, frames and bytes by kind as rates with drops by cause and each group's gossip figures | agentspaces.console.*, a read-only peer |
The arithmetic for reliability is worth internalizing. Work that any one of N workers can do has availability 1 - (1 - a)^N, so three workers at 99% give six nines and more workers add almost nothing. A decision that needs a majority of N has a harsher curve: three voters at 99% give three and a half nines, five give five. Past three or four of a kind the floor takes over: the model provider all workers call, the seed every peer dialed, the ordered log's majority set. Diversify the floor (two providers, two seeds, log members on separate hosts) before adding agents.
Three caveats worth noting.
- There is no universal cutoff at three or four. Every additional independent worker in the any-one case reduces the failure probability by 100 times. Whether that helps materially depends on the shared dependencies' failure rates and your target.
- Availability and correctness are different. Having a majority online does not establish that an agent's decision is correct. The analogous voting calculation requires independent errors and a decision rule under which a correct majority produces a correct result.
- Diversity must remove the actual shared failure. Two providers help only if failover works; separate hosts may still share power or networking. A seed used only during initial discovery may limit bootstrap availability without limiting ongoing operation. AWS's reliability guidance says the same.
The consistency ladder
The rungs
Each space promises one thing about who may complete an entry, and the promise has a price. The library's design rule is escalation: a space starts on the lowest rung that is correct for its workload and climbs one rung when the workload proves it must. Nothing on a lower rung has to move when one stage climbs, because the rung is a property of the space.
| Rung | The promise | The protocol | The cost | What it leaves open |
|---|---|---|---|---|
| Idempotent write | a repeat changes nothing | a deterministic entry id; the CRDT folds the duplicate | none | nothing, for stages that can be written this way |
LEASE_RACE take | one holder per lease period | a signed claim, a settle window (200 ms by default, about two gossip periods), the lowest (stamp, peer) wins | one settle window of latency | a transient duplicate across a partition or a back-dated claim |
AUCTION | the cheapest holder | claims carry a price from a bid function; convergent bidding in a few rounds | a few rounds; one bid function per space per peer | a lying bidder; prices are untrusted |
LEASED ticket | once per key, per lease | a join or reduce key serialized by a leased ticket taken under the race | one claim per key | a duplicate under partition |
ORDERED log | exactly one completion, fleet-wide | a Raft log commits each claim; @OrderedTake, ORDERED joins and reduces | a quorum round trip per claim; a fixed member set; a live majority | nothing about external systems: make the irreversible call last, keyed by the entry id |
When to climb
Climb from the first rung when a stage has a side effect you cannot make idempotent. Climb from the lease race to an auction when workers are not interchangeable and the choice among them is a cost you can compute. Climb to a leased ticket when a stage fires once per key and keys can arrive out of order (a join, a running total). Climb to the ordered log when a duplicate costs money, safety, or trust, and accept the quorum: pick the three or five peers with the steadiest uptime for the member set, host the log on the space the stage completes in, and keep the irreversible external call as the last step, keyed by the entry id, so that a retry after a crash finds it done. The Party Bus keeps one of its nine spaces on the ordered log; example 18 keeps one of five. Most stages stop at the second rung.
Deciding: votes, quorums, estimates
A fleet decides in two ways, and the architecture is which one each decision gets. A vote is a space of signed ballots: a proposal names its options and quorum, each eligible voter writes one ballot, and the decision closes when the tally reaches the quorum, recomputable by any replica from the space itself. An estimate is a push-sum aggregate: each member contributes a number, members exchange halves of their mass every tick, and every member converges on the fleet-wide sum, average, count, extremum, or quantile. The vote is slow, auditable, and bounded by an electorate; the estimate is fast, approximate, and open.
| Use a vote when | Use an estimate when | Use a workflow join when |
|---|---|---|
| the answer must be defensible later | the answer is a number that a few percent of error cannot hurt | the answer is the presence of several records about one case |
| the electorate matters (grants, agent-attested voters, one vote each) | every member's contribution counts equally and nobody can be excluded | the parts arrive from different stages and the fan-in fires once per key |
| money, safety, or policy turns on it | load, backlog, counts, thresholds, readiness | the stage is data-driven rather than opinion-driven |
Three decisions shape a vote. Granularity: one peer is one vote by default; name peer/agent in the vote grant and the count becomes one attested agent one vote, which is what lets a council of several agents sit on one peer. Quorum: too low and one peer decides, too high and a lost peer stalls the vote until its lease lapses; three of five is the usual answer, and an odd electorate avoids ties (a tie closes to the lexicographically first option, so avoid relying on it). Who opens it: a stage returns a Motion when its question, options, or quorum depend on the cue, or carries @Propose when the shape is fixed and only the key varies; either way the proposal opens once per id however many replicas see the cue.
Two decisions shape an estimate. What settles it: a settle rule fires when the estimate has held for N steps within a tolerance, a step being a gossip round the node took part in once it has heard the epoch, or the epoch going quiet, so a node nobody pushes to never settles on its own value and a plateau before a late contribution is not settled; pick N from the fleet's diameter and the spread of the contributions (example 18 needs forty at a 250 ms tick, since one reserve follows the ordered log) and the tolerance from what the consumer can tolerate. Who takes part: shares go only to members whose aggregate advertisement is live, so a console or a client that has not started the capability never swallows mass; a member that must count, must advertise. An estimate does not detect a poisoned contribution; the moment one would matter, the decision is a vote.
When a workflow, and how many spaces
A workflow on this fabric is nothing more than records that carry a case id moving through spaces, each stage a binding that takes or reacts and writes the next record. That is why there is no engine: the vocabulary of the workflow guide (sequence, map, fork, fan-out, join, gather, reduce, compose, filter, route, branch, decide, exactly once, deadline, loop, compensation, the human step, the model step, the planner) is each a way of binding, and the engine's state is the spaces themselves. Reach for it when a business case passes through several hands and each hand must survive a crash; keep a single take when one worker does the whole job.
| Shape | When | What to watch |
|---|---|---|
| One space, one record type per stage | a short pipeline whose stages are all takes and all workers read the lane | every stage's lease fits its work; the lane's strategy fits every stage |
| Several spaces, one group | stages differ in strategy, admission, or retention; a join gathers parts from several lanes; one stage needs the ordered log | a part may live in another space; an ORDERED join lives on the space that hosts the log; reactions fire on every replica, so what must happen once is a take, a fleet-wide join, a vote, or an idempotent write |
| Several groups | the parties differ: a tenant's flow hands a commitment to a partner's, a conversation group reports to a mission group | a bridge peer in both groups, bound by group name; the bridged record is the contract; no shared ordered log across groups, so the bridge takes once on one side and writes idempotently on the other |
Deadlines are leased reminders: a stage writes a reminder with the lease it is willing to wait, and a reaction on the lease's expiry escalates; the reaction fires within a tick of the lapse and nowhere else. Loops are stages that write their own cue again with a bounded counter. Compensation is a stage like any other, written when a later stage fails, taken by the agent that can undo. A planner or a model is a stage whose implementation is a model call; the take's lease is renewed on every model round-trip by the host libraries, so a long think does not lapse the task.
Deployment recipes
| Recipe | Topology (ch. 1 to 3) | Boundaries (ch. 4, 5) | Guarantees (ch. 6 to 9) | Edge |
|---|---|---|---|---|
| Edge fleet | 3 to 15 listening JVM peers on one network, every peer a seed, mtls | one group, one space per lane | lease race everywhere; three workers per kind; a vote where a human would have signed | a console peer |
| Hub and field | a few listening seeds in the hub, many dial-only field peers (laptops, boxes, Python and TypeScript clients), a relay for field-to-field | one group; field lanes admitted by allowlist | the hub holds the ordered log; field work is lease-race takes with long leases | a console and, for partners, an A2A gateway in the hub |
| Multi-site | a rendezvous per site, relays across, one group | spaces per lane; a content key if the sites' links are not trusted | ordered log members on the site with the steadiest links; settle windows sized to the WAN | a gateway per site if partners dial in |
| Partner integration | the partner runs peers of its own or dials an A2A gateway | two groups and a bridge peer; the partner group under POLICY or INVITE; a content key per group | commitments on an ORDERED space in the group that owns them; idempotent writes across the bridge | A2A for agent protocols, MCP for model tooling |
| Polyglot workers | a JVM coordinator with Python and TypeScript workers dialing in | one group; the workers' lanes admitted by allowlist; dev-local or a TCP-fronting peer | clients take under the lease race and contribute to estimates and votes; once-only stages stay on the JVM | the trilingual demo's council is the template |
| Human in the loop | any of the above plus a console peer with the directive grant | an audit space for every command | approvals as votes with the panel in the grant; deadlines as leased reminders that escalate | the console's command-and-control: dispatch, cancel, pause, resume, drain |
The architect's checklist
- Peers. Each peer's purpose is one of worker host, coordinator, rendezvous, relay, console, gateway, or bridge, and the count of each follows chapter 1's table. Every worker kind has three or four instances on different peers.
- Clock. The tick and gossip period are chosen, documented, and the same for every member; every lease, settle window, and deadline is a stated multiple of them.
- Bootstrap. Every peer has at least two seeds; each site has a rendezvous; dial-only peers that must reach each other have a relay. The founding path (starter string, or a kept founder key in code) is decided with the revocation and rotation paths it implies.
- Profiles. Every peer runs a security profile that meets the others'; where a non-JVM client dials in, the TCP listener it reaches is deliberate and
require-attestationis off only where that is intended. - Lanes. Each space has one strategy, one admission rule, one retention, and a reason for existing; records carry the case id; references are ids.
- Perimeters. Each group has a membership policy, a content-key decision, and an authorizer with named grants; any bridge peer is listed with the records it carries across.
- Rungs. Each space's rung on the ladder is stated with the workload fact that justifies it; ordered logs have a named member set of three or five and the irreversible call is last.
- Decisions. Each decision is a vote or an estimate with the reason; votes have a quorum, a granularity, and an electorate; estimates have a settle rule and only advertising participants.
- Failure. For each stage: what lapses, who picks it up, what a reaction drops; for each partition: which side keeps working and which stops.
- Observation. A console peer exists; the operator can name the written, completed, queued, and in-progress counts per space and the suspect state per peer; every command is audited.
- Proof. The flow has a test that reads every property from a space, runs under a crash, and (where languages mix) runs the workers in their own processes.
RELIABILITY.md for the availability arithmetic, and SPEC.md and TECH-SPEC.md for the formats and the rationale.