aspace://architecture/start

How to read this guide

This guide is for the person who decides how a system on AgentSpaces is shaped: how many peers, where they run, which boundaries the application draws, what each space promises, and where the fleet meets the rest of the enterprise. It assumes some working knowledge of distributed systems. It's a companion of the developer guide for the API and the workflow guide for the stage-by-stage patterns; this guide decides what those two build.

We designed AgentSpaces to bookend a story that began at Bell Labs in 1986 with the publication of Linda, continued on through the partnership with Sun Microsystems and their release of Jini and JXTA. When we arrived at Lucent in 96 and '00, we reviewed, designed, and prototyped with Jini and JXTA, though didn't wind up including it on any of the product lines we architected. We're ultimately coming full circle as the Bell Labs crew comes back together again at Bad Monkey to combine the best parts of Jini and JXTA with a healthy dose of modernization on related standards (PKI, TLS, mTLS, DTLS, CBOR, etc). While P2P found a home in CDNs and other large-scale economic saving endeavors, technical improvements have been driven by a steady state of research in the intervening years. If we implement one of those papers, we'll cite it in the documentation.

What AgentSpaces is, in one sentenceA fabric of signed, leased entries replicated across the peers of a group, with coordination services on top of that replication, so that agents on any host, in any language, coordinate through shared time-boxed memory rather than through a broker or a database. Everything in this guide follows from three properties: every entry is signed and leased, every peer holds a full replica of the spaces it registers, and every verb is a protocol the peers run among themselves.
aspace://architecture/picture · chapter 0

The system in one picture

ONE GROUP: EVERY PEER HOLDS A REPLICA OF EVERY SPACE IT REGISTERSpeer-aSpring Bootintakedecisionsledgeragents, cardscapabilities on pipespeer-bMicronautintakedecisionsledgeragents, cardscapabilities on pipespeer-cPythonintakedecisionsledgeragents, cardscapabilities on pipespeer-dconsole, read-onlyintakedecisionsledgeragents, cardscapabilities on pipesgossipgossipgossipsigned entries, leases, anti-entropy, SWIM membership, one tick clockWHAT A PEER IS MADE OFidentitypeer key, certificatespeeringtransports, SWIM, gossipdiscoverycards, ads, semanticspacesreplicated tuple spacescapabilitiesvote, aggregate, log, learnthe programming model: annotations on plain objects, bound by AgentBinder; the same verbs from Python, TypeScript, and Clojure
The system in one picture. A group is a set of peers that gossip; every peer that registers a space holds a full replica of it, so there is no broker, no primary, and no database between agents. A peer is five layers under a programming model; a Python or TypeScript peer is the same wire with the same layers up to the ordered log.

A peer is one process running the library. It has an Ed25519 identity, joins one or more groups, and registers the spaces it takes part in. A group is the set of peers that gossip with one another: membership by SWIM probes, advertisements by a gossip stream, space state by anti-entropy. A space is a replicated tuple space: typed entries with leases, written, read, taken under a lease, and completed, all as signed records folded into a CRDT so that every replica converges whatever order the records arrive in. Capabilities (votes, push-sum aggregates, the ordered log, semantic discovery, gossip learning) run on pipes between peers of the group. Agents are ordinary objects bound to spaces and capabilities by annotations, or by the equivalent decorators in Python and TypeScript and the defagent form in Clojure; each publishes a signed card the others discover.

Four consequences shape every architecture on it. There is no primary and no broker, so there is nothing to fail over and nothing to scale separately; a peer that registers a space holds all of it. Convergence has a rate, set by one tick clock per peer (250 ms by default) and a gossip period (1 s) carried in the group's advertisement: a write reaches every replica in a few periods, and the settle windows, leases, and deadlines of chapters 6 to 9 are multiples of that rate. A guarantee stronger than replication (one taker per entry, one completion fleet-wide, a decision) is a protocol run on demand on the spaces that need it, so each space stands on its own rung of the ladder in chapter 7. And identity is on the wire: each record carries its issuer's signature, so the trust perimeter is membership and admission rather than a network edge.

Where the pieces are describedThe layers and their APIs: the developer guide's chapters on the Space API, discovery, capability services, peering and transports, and security. The formats and algorithms: SPEC.md and TECH-SPEC.md (section 2 of the technical specification, "Why these protocols", is the rationale this guide builds on).
aspace://architecture/sizing · chapter 1

How many peers, and what runs where

Count peers by what they hold; the number of agents is a separate question. An agent is a binding on a peer, and a peer can host as many agents as its process can run; three peers can seat thirty agents. A peer is a unit of replication (a full copy of each space it registers), a unit of failure (its take leases lapse together), a unit of trust (one identity, one keystore, one channel), and, for the ordered log, a unit of quorum. Decide the peer count on those four axes and let agent placement follow.

Three sizes

SizeWhat it looks likeWhat changesWhere it is shown
3 to 5 peersone site, every peer listens, every member is a seednothing special: full views, full replicas, one ordered log needs a majority of 2 of 3 or 3 of 5examples 02, 12, 18 (three peers, five spaces); the Spring, Micronaut, and LangChain4j example fleets
5 to 15 peersa site or two, a rendezvous, roles emerge (console, gateway, relay)advertisement caches fill (8,192 entries by default, 32,768 on a rendezvous); keep card TTLs and refresh at the defaults; decide which few peers hold the ordered log's member setthe Party Bus: 15 peers, 9 spaces, 34 agents on TLS
Beyond thatseveral sites, many dial-only workers, partner groupsreplication is full by default, so put a large space on a tag-sharded replica only on the peers that need it (builder-only today); split by group rather than by space when the perimeter differs; expect no published scale ceiling, measure with the stress harnesses in agentspaces-perf and the traffic auditsa fifty-member simulated fleet in the peering suite (RobustnessScaleTest) idles at about five frames per member per tick and drops a crashed member from every view within its TTL; six real peers idle at 0.7 frames per peer per second, the same across two hosts (examples/superfriends/two-host-audit.sh, PERF2 ยง5)

The library publishes no peer-count limit. Three things grow with the fleet: membership views (full; every peer knows every member), replication (full per registered space; every replica holds every entry until its tombstone collects), and the advertisement cache (bounded, evicting on read). Gossip propagation is O(log N) rounds, and a per-issuer token bucket (50,000 frames, 10,000 a second) bounds what any one peer can push. The one cost that is linear per member is the advertisement refresh, which goes directly to every member: once a fleet outgrows what the probe cycle can keep alive inside the member TTL, the refresh runs at half the TTL and is the keepalive, so an idle member costs about (N - 1) frames per refresh interval; probes and anti-entropy back off while nothing moves. None of these is a cliff at ten or fifty peers, but each is a reason to keep a group to one purpose and to use several groups for several purposes.

Roles

Two roles exist on the wire, and three more are conventions you adopt. RENDEZVOUS is a peer that keeps a large advertisement cache and answers discovery queries directly; a site seeds from it. RELAY is a peer that forwards a fully signed frame from one member to another, one hop only, so that two dial-only peers can reach each other; relays never re-relay. By convention a console peer is a read-only member that serves the operator's view and, with the directive grant, command-and-control; a gateway peer fronts the fleet for A2A or MCP clients; a coordinator peer holds an ordered log's leadership most of the time, though any member of the log's set may lead.

DecisionRuleWhy
Workers per kindthree or four, on different peersone-of-N availability stops improving past that point and the floor (shared provider, shared seed) takes over; see chapter 6
Ordered logsone per group per peer, a fixed member set of three or fivea quorum is a majority; reconfiguration means a new log; keep the set on the peers with the steadiest uptime
Rendezvousone per site, two where the site must not lose discoveryit is a cache, never an authority; losing it slows discovery and changes no result
Console and gatewaytheir own peers, read-only or edge-facingthey hold no take leases, so their restarts cost the flow nothing, and their inbound ports are the only ones you expose
Dial-only peersas many as you likea peer with no bind advertises no endpoint, speaks only to what it dialed, and needs a relay to speak to another dial-only peer
Keep in mindEverything runs on one clock, somewhat inspired by game engines and the notion of a tick of the clock. The clocks are injectable, and we change them in the test harness to speed things up in certain cases. The tick drives the SWIM probe, the self-advertisement, anti-entropy, and every capability's protocol round; the gossip period is carried in the group advertisement and is the same for every member. Changing either changes the convergence rate of the whole fabric, and with it the right settle windows and deadlines. Typically the default should be left as is, but you can set them once, early, and derive the rest.
aspace://architecture/discovery · chapter 2

Discovery and bootstrap

A. ONE SITE, FULL MESHpeerpeerpeerpeerpeer3 to 15 peers; seeds are any memberB. SEEDS AND DIAL-ONLY PEERSseedlistens, tcp or tlslaptopno bindpythonno bindfield boxNATrelayrole RELAYdial-only peers reach each other through a relayC. TWO SITES, RENDEZVOUSrendezvousrole RENDEZVOUSsite 1peerpeersite 2peerpeereach site seeds from the rendezvous;a cache, never an authority
Three topologies that cover most deployments. A: one site, every peer listens and any member is a seed; views and replicas are full. B: a few listening seeds and many dial-only peers (laptops, Python and TypeScript clients, boxes behind NAT); a RELAY role lets two dial-only peers reach each other in one hop. C: sites behind their own networks seed from a RENDEZVOUS peer, which is a large advertisement cache and nothing more; losing it slows discovery and breaks nothing.

The options

A peer joins a group in one of two ways. With a founding string, every member derives the same group id locally from the same document, so any member is a seed and nobody holds a secret; this is what the starter's founding: does and what every example uses. With join by group id, the peer dials a seed, asks for the group's founding document, verifies that the document hashes to exactly the id it wanted, and only then introduces itself; a hostile seed can delay that join and can never substitute a policy. The first is right inside one organization; the second is right when the id travels further than the document (a partner's configuration, a printed credential, a registry).

MechanismWhat it givesWhat it costsWhen
Seeds (groups[].seeds)a way in; a member once joined needs them no morea list to maintain; a seed that is down is only a delay if there is anotheralways; give every peer at least two
Rendezvous rolea large advertisement cache that answers queries directly and seeds a whole siteone more peer to run; none to correctnessa site, a WAN, any fleet past a handful of peers
Relay roleone signed hop between two peers that cannot dial each othera relay peer per region; relayed frames are refused under required attestationdial-only peers that must reach each other; NAT on both ends
Multicast beacon (agentspaces.multicast.*, off by default)LAN peers find a seed without configurationUDP on the segment; only a hint under required attestationa lab, a vehicle, a floor of devices on one segment
Join by group ida verified founding document before the first framea reachable seed that joined with the signed document; a timeout (10 s by default)ids that cross an organizational line

There is no universal bootstrap group and no mDNS; a deployment that needs a directory of groups founds one and seeds from it. The rendezvous and relay roles are declared by the peer that plays them (agentspaces.roles) and learned by everyone else from its advertisement.

Cards and advertisements

Every discoverable thing is a signed, typed, time-limited document: a peer advertisement (TTL 10 minutes), an agent card, a space advertisement, a capability advertisement (15 minutes each), the group advertisement (24 hours). They travel on one gossip stream, every cache keeps one per id and replaces it only with a newer issue, and a refresh cadence (card-refresh-millis, 5 minutes by default) keeps live things live. Revocations travel the same way and are never cached: what a replica already holds stays, and what arrives after is refused. Semantic discovery ranks cards by an embedding (a deterministic hashing embedder by default, a real model when you give one) and asks up to three peers over a pipe for theirs; it is a routing hint, never a roster.

The design rule that follows: route work by entry type and treat a discovered name as a hint. An agent card tells a planner what is available and a remote caller how to address it; the space and the record type are what actually deliver the work, and they do so whether or not the card has arrived yet. A fleet whose correctness depends on a discovery result has put an eventually consistent cache on its critical path.

The starter's founding, and what it reservesA group founded through the starter is OPEN, uses the default gossip parameters and SWIM configuration, and is founded by a generated key the starter does not keep. Operations the specification reserves to the founder, revoking a peer and rotating the content key without a grant, therefore need either a group founded in code with a kept founder key (PeerNode.joinGroup(SignedGroupAdvertisement, ...)) or the grants the starter does expose (security.grants.key-rotator, the founder-or-ca revocation validator). Decide which before the fleet has data in it. The membership policies INVITE and POLICY are likewise hand-wired today. See the developer guide's security chapter.
aspace://architecture/mixing · chapter 3

Mixing peers: hosts, languages, profiles

One group may hold peers on every host the project supports: the Spring Boot starter, Micronaut, LangChain4j on a plain JVM, Embabel, a peer built from the core API, and the Python, TypeScript, and Clojure clients. They share one wire (CBOR, Ed25519, wire version 3, pinned by golden vectors every client runs), so the question is never whether they can talk but what each can host. Decide placement by the table, then write each agent in the language its team writes.

Peer kindHostsDoes not hostPosture
JVM host (Spring, Micronaut, LangChain4j, Embabel, core)every space strategy, every capability including the ordered log, A2A and MCP gateways, the console, model servinglistens or dials; TLS by default (mtls)
Clojure (agentspaces-clj)the same as a JVM host through fleet/start and defagent, including ordered takes and joinsthe same
Python clienttakes, notifies, bids, ballots, decisions, proposals, estimates, LOCAL joins, push-sum, semantic queries, remote actions, its own agent certificatethe ordered log, so no ordered takes, reduces, or fleet-wide join modes; gossip learningdial-only on plain TCP; sends capability frames only to members it can reach
TypeScript clientthe same as Pythonthe samethe same

Three rules keep a mixed fleet coherent. Profiles must meet. Note that in 0.3.0, a non-JVM client speaks plain TCP, so the peer it dials must listen on TCP: either the fleet runs dev-local (signed frames, no channel encryption) or a JVM peer fronts the clients with a TCP listener while the JVM core stays on TLS; a mixed fleet degrades to its lowest rung unless require-attestation refuses the plaintext link outright. We expect 1.0.0 to provide the same transport options universally across all libraries, we started with Java so it proves the architecture and then we uplift python, typescript, etc.Once-only work lives on a JVM peer. A Python or TypeScript worker takes under the lease race and completes atomically like any other, but a stage that needs the ordered log binds on a JVM host, and the clients write into and read from that space. Capabilities announce themselves. A peer that runs an aggregate advertises it, and shares go only to members whose advertisement is live, so a bare client or a console never swallows mass; a client that wants to contribute starts its aggregate and advertises like everyone else.

The proofCouncilFleet in example 02 runs a Java coordinator with Python and TypeScript council workers in their own processes: tasks taken, a vote decided at a quorum of three, a push-sum epoch settled at the fleet mean, a semantic query answered over the pipe, a remote action served. run-trilingual-demo.sh plays it as its second act. The developer guide's language chapter has each client's surface.
aspace://architecture/boundaries · chapter 4

Spaces as lanes, groups as perimeters

GROUP: CLAIMS (ONE TENANT, ONE MISSION)intakeLEASE_RACEassessmentAUCTIONdecisionsvotespaymentsORDERED logledgerreduce accumulatorsone strategy and one retention per space:a lane is a space, as a queue would beteam A peersregistrars, screensteam B peersadjusters, panelops peerconsole, gatewayadmission per space: GROUP, ALLOWLIST, CREDENTIAL, AUTHORIZERGROUP: PARTNER (ANOTHER ORGANIZATION)commitmentsORDERED, content key sealedcatalogcards and assetspartner peerstheir hosts, their keysbridge peera member of both groupsgroup=claimsgroup=partnermembership policy OPEN, INVITE, or POLICY (DID)
Spaces draw lanes, groups draw perimeters. Inside one group, give each lane of the application its own space, since strategy, admission, and retention are per space. Across tenants, missions, or organizations, use groups: membership is the trust perimeter. A peer may belong to several groups and bind an agent into each by name, which is how one organization's flow hands commitments to another's without sharing a replica of everything.

One space per lane

A space carries one strategy (LEASE_RACE or AUCTION; the ordered log is a coordinator registered on a space rather than a third strategy), one admission rule, one replication mode, one settle window, and one retention policy through its leases. So the unit of design is the lane: give each stage of the application its own space the way you would give it its own queue, and let the record types say what flows. Example 18 runs a nine-stage claims flow over five spaces (intake, assessment, decisions, payments, ledger), each with the strategy its stage needs: the assessment space is an auction because adjusters price work, the payments space hosts the ordered log because a payment happens once, the ledger holds reduce accumulators because exposure is a running total.

Draw a new space whenKeep one space when
the strategy differs (one lane races, another auctions)the records are stages of one case and the same workers read them
admission differs (a lane only an attested agent may write)a reader wants one template over the whole lane
retention differs (hours for tasks, days for an audit trail)a join draws parts from them (a part may name another space; the join's own space holds its tickets)
the ordered log must live there (one log per peer per group)the volume is small enough that a full replica everywhere costs nothing

Within a space, tags and filters partition without a new space: a template matches on tags before it decodes a payload, so a keyless relay can route by tag, and a where filter narrows a worker to the entries addressed to it. Example 18 deploys one shop class per region on one space. Reach for a tag shard (TAG_SHARDED replication, builder-only today) when a space is large and most peers need one slice of it.

A group per tenant, mission, or conversation

A group is the membership perimeter: who gossips, who sees the advertisements, whose signatures count. One peer may belong to several, and an agent binds into a named group (group = "..." on every annotation, or every satisfying group when unnamed) with one card published per group. Draw a new group when the set of parties differs: a tenant, a mission with its own lifetime, a partner organization, a conversation that should leave no trace in the others. Draw a new space when only the lane differs. The bridge between two groups is a peer that is a member of both and hosts an agent in each; it hands records across by taking in one and writing in the other, and it is the one place a cross-group policy can be enforced.

BoundaryMechanismStarter support
Who may joinmembership policy OPEN, INVITE (a founder-signed credential), POLICY (a validator SPI, for instance DID)OPEN; the other two are wired in code
Who may write or take a spacespace admission GROUP, ALLOWLIST, CREDENTIAL, AUTHORIZER, judged per acting agent; reads stay open to the groupspaces[].admission, allowed-agents, credential-issuer
Who may read a payloada group content key (AES-256-GCM); members without it relay and cannot read; rotation with a cutover delaygroups[].content-key and content-key-rotation
Who may do an operationthe authorizer's nine operations under security.grants.*, or OIDC scopesyes
Keep in mindTypes, tags, issuers, and leases stay plaintext even under a content key: the fabric routes on them. If the metadata itself is sensitive, the boundary is a group; a key hides payloads alone.
aspace://architecture/sharing · chapter 5

Sharing across teams and organizations

Sharing inside one organization is a question of lanes and admission; sharing across organizations is a question of trust roots. The fabric gives you four independent levers, and the architecture is which of them you pull for which boundary.

LeverWhat it decidesGranularityWhere
Membershipwho is in the group at allpeerthe founding document's policy; INVITE credentials; a POLICY validator
Admissionwho may write or take a given spaceacting agentthe space advertisement; allowlists and credentials
Authorizationwho may vote, take, write, serve a model, rotate a key, issue a directivepeer, or peer/agent with attested agentsMembershipAuthorizer grants, or an OidcAuthorizer with aspace:<op> scopes
Confidentialitywho may read a payloadkey holdergroup content keys, served by key-wrap, rotated on a schedule

Two teams in one organization usually share a group and separate by space and admission: team A's registrars write intake, team B's adjusters take assessment, both read decisions, and the authorizer's VOTE grant names the panel. Two organizations usually get two groups and a bridge: each keeps its own founder, keystore, and content key; the bridge peer is a member of both, admitted on each side under that side's policy, and the records it carries across are the contract between them. Where a partner should see one lane and nothing else, give that lane its own group rather than its own space, since reads in a space are open to the whole group.

Identity is what makes the levers compose. Every record carries its issuer's signature, so a replica can judge admission and grants without asking anyone. An agent can act under its own certified key (agent-attested) rather than its peer's, which is what lets a grant name one agent on a shared peer and what the AGENT vote granularity counts. Revocation is eventually consistent and follows a freeze rule: what a replica holds stays, what arrives after the revocation is refused, and a key-compromise revocation refuses everything new at once. Plan the revocation path before the first partner joins: who issues it (the founder, or a CA under founder-or-ca), how it reaches every replica (gossip, within a few periods), and what the frozen history means for the records already shared.

The cross-organization blueprintA POLICY group with DID credentials for membership, a group content key with key-wrap so a relay can carry what it cannot read, an ORDERED space for the commitments both sides must see once, and an A2A gateway for partners that run no peer at all. The use-case models on the site sketch it; the pieces each have a chapter in the developer guide.
aspace://architecture/reliability · chapter 6

Building up reliability

A TAKE LEASE IS THE RECOVERY CLOCKwrittentaken, lease 2mworker killedlease lapsesreappearscompleted by a siblingnothing is retried by a scheduler: the lease lapses and the entry is simply available againREDUNDANCY IS A COUNT, THEN A FLOOR3 workers at 99%1-of-N: six nines3 voters at 99%majority-of-N: 3.5 nines5 voters at 99%majority-of-N: 5 ninesafter three or four workers per kind the count stops mattering and the floor does:the model provider every worker shares, the one seed everyone dialed, the one peer that holds the ordered log's majority.Diversify the floor before adding agents.
Reliability comes from leases and counts. Work survives a crash because a lease lapses and the entry reappears for any sibling that holds the same binding; there is no retry queue to operate. Availability then follows the arithmetic: one-of-N work needs few workers, majority-of-N decisions need an odd count and a live majority, and past three or four of a kind the shared dependencies set the floor.

Reliability on this fabric is built from four mechanisms you compose rather than one you switch on. The lease is recovery: a take holds an entry for a declared time, and a worker that dies lets it lapse, after which the entry is available to any sibling; nothing schedules a retry and nothing has to notice the death. The count is availability: bind the same worker on several peers and the lane drains while any one lives. Replication is durability within the group: an entry lives on every replica from its first gossip period, and a crashed peer's replica is rebuilt by anti-entropy when it returns. And the ordered log is the one place a majority is required, so it is the one place a partition costs availability.

QuestionAnswerKnob
How long until a crashed worker's task is retried?the take lease, plus one ticklease on @SpaceTake: 30 s for a quick task, 2 m for a model call, 10 m for a long job; always longer than the work
How long until a crashed peer is forgotten?the member TTL (30 s by default)SWIM configuration, in code
What if a link drops, or two members never met?a JVM peer retries a lost member's last endpoints and re-dials its seeds when its view is empty, backing off to thirty seconds; the view itself is anti-entropy state, so a member whose introduction was lost is learned from any partner; a Python or TypeScript peer re-dials its seed, re-introduces itself, and re-pulls what it hadnothing to configure; ROBUSTNESS.md has the scenarios
What happens during a partition?each side keeps writing to its replicas and they merge on heal; a lease race may complete the same entry on both sides once; the ordered log's minority side cannot takewhich spaces stand on the ordered log (chapter 7)
What does a reaction do on failure?nothing: a @SpaceNotify that throws is logged and droppedput work that must happen behind a take; put nice-to-have reactions behind a notify
How do I see it?the console: per space the written, completed, queued, and in-progress counts; per peer the suspect state and the channel mode; the wire-traffic panel, frames and bytes by kind as rates with drops by cause and each group's gossip figuresagentspaces.console.*, a read-only peer

The arithmetic for reliability is worth internalizing. Work that any one of N workers can do has availability 1 - (1 - a)^N, so three workers at 99% give six nines and more workers add almost nothing. A decision that needs a majority of N has a harsher curve: three voters at 99% give three and a half nines, five give five. Past three or four of a kind the floor takes over: the model provider all workers call, the seed every peer dialed, the ordered log's majority set. Diversify the floor (two providers, two seeds, log members on separate hosts) before adding agents.

Three caveats worth noting.

  • There is no universal cutoff at three or four. Every additional independent worker in the any-one case reduces the failure probability by 100 times. Whether that helps materially depends on the shared dependencies' failure rates and your target.
  • Availability and correctness are different. Having a majority online does not establish that an agent's decision is correct. The analogous voting calculation requires independent errors and a decision rule under which a correct majority produces a correct result.
  • Diversity must remove the actual shared failure. Two providers help only if failover works; separate hosts may still share power or networking. A seed used only during initial discovery may limit bootstrap availability without limiting ongoing operation. AWS's reliability guidance says the same.
The honest limitsA lease race can complete the same entry once on each side of a partition; if that is unacceptable, the stage climbs to the ordered log (chapter 7). A reaction that must not be lost is a take. There is no durable workflow history and no saga engine: a saga is explicit request, success, failure, and compensation records, each leased, each a stage (workflow guide, chapter 10).
aspace://architecture/ladder · chapter 7

The consistency ladder

idempotent writea deterministic entry id; a repeat is a no-opcost: freeleaves open: duplicates foldLEASE_RACE takesigned claim, settle window, lowest (hlc, peer) winscost: one gossip windowleaves open: a transient duplicate across a partitionAUCTIONclaims carry bids, the lowest price winscost: O(diameter) roundsleaves open: lying bidders; untrusted pricesLEASED ticketa join or reduce key serialized by a leased ticketcost: one claim per keyleaves open: a duplicate is possible under partitionORDERED loga Raft log commits the claim; exactly one completioncost: a quorum round trip per claimleaves open: a minority partition cannot takeclimb one rung at a time, when the workload proves it must; most stages stop at the secondstronger guarantee, higher cost
The consistency ladder. Each rung buys a stronger once-only guarantee at a higher cost and a new failure mode. An idempotent write needs nothing; a lease race needs one settle window; an auction converges in a few rounds and trusts its bidders; a leased ticket serializes a key without a quorum; the ordered log is exactly once at the price of a live majority. Nothing on a lower rung has to move when a stage climbs.

The rungs

Each space promises one thing about who may complete an entry, and the promise has a price. The library's design rule is escalation: a space starts on the lowest rung that is correct for its workload and climbs one rung when the workload proves it must. Nothing on a lower rung has to move when one stage climbs, because the rung is a property of the space.

RungThe promiseThe protocolThe costWhat it leaves open
Idempotent writea repeat changes nothinga deterministic entry id; the CRDT folds the duplicatenonenothing, for stages that can be written this way
LEASE_RACE takeone holder per lease perioda signed claim, a settle window (200 ms by default, about two gossip periods), the lowest (stamp, peer) winsone settle window of latencya transient duplicate across a partition or a back-dated claim
AUCTIONthe cheapest holderclaims carry a price from a bid function; convergent bidding in a few roundsa few rounds; one bid function per space per peera lying bidder; prices are untrusted
LEASED ticketonce per key, per leasea join or reduce key serialized by a leased ticket taken under the raceone claim per keya duplicate under partition
ORDERED logexactly one completion, fleet-widea Raft log commits each claim; @OrderedTake, ORDERED joins and reducesa quorum round trip per claim; a fixed member set; a live majoritynothing about external systems: make the irreversible call last, keyed by the entry id

When to climb

Climb from the first rung when a stage has a side effect you cannot make idempotent. Climb from the lease race to an auction when workers are not interchangeable and the choice among them is a cost you can compute. Climb to a leased ticket when a stage fires once per key and keys can arrive out of order (a join, a running total). Climb to the ordered log when a duplicate costs money, safety, or trust, and accept the quorum: pick the three or five peers with the steadiest uptime for the member set, host the log on the space the stage completes in, and keep the irreversible external call as the last step, keyed by the entry id, so that a retry after a crash finds it done. The Party Bus keeps one of its nine spaces on the ordered log; example 18 keeps one of five. Most stages stop at the second rung.

The things in betweenTag shards (a partial replica of a large space), filters and tags (a worker that sees only its slice), settle windows (longer where gossip is slower), and admission (fewer writers) are all ways to narrow a lane without climbing a rung. The developer guide's conflict strategies and the workflow guide's exactly-once chapter work through the choices.
aspace://architecture/deciding · chapter 8

Deciding: votes, quorums, estimates

A VOTE: QUORUMProposalid, question, options, quorumMotion or @ProposeBallotsigned, one per voterBallotsigned, one per voterBallotsigned, one per votertally at quorumDecisionwinner, tally, granularityauditable, recomputable from the space; one peer or one agent is one voterevoke a voter and the count reopensAN ESTIMATE: PUSH-SUMpeervalue 10peervalue 30peervalue 20halves exchanged each tickestimate 20.0settled: unchanged for N heard ticksapproximate, fast, no electorate; shares go only tomembers that advertise the aggregate; a silent nodenever settles on its own valueuse for sensing; never for a trust-sensitive decision
Two ways to decide, for two kinds of question. A QUORUM vote is a space full of signed ballots: slow, auditable, and the right tool when the answer must be defensible. A push-sum estimate is a converging number every member ends up holding: fast, approximate, and the right tool for sensing load, counts, and thresholds. The ladder's rule applies here too: an estimate that gates money or safety becomes a vote.

A fleet decides in two ways, and the architecture is which one each decision gets. A vote is a space of signed ballots: a proposal names its options and quorum, each eligible voter writes one ballot, and the decision closes when the tally reaches the quorum, recomputable by any replica from the space itself. An estimate is a push-sum aggregate: each member contributes a number, members exchange halves of their mass every tick, and every member converges on the fleet-wide sum, average, count, extremum, or quantile. The vote is slow, auditable, and bounded by an electorate; the estimate is fast, approximate, and open.

Use a vote whenUse an estimate whenUse a workflow join when
the answer must be defensible laterthe answer is a number that a few percent of error cannot hurtthe answer is the presence of several records about one case
the electorate matters (grants, agent-attested voters, one vote each)every member's contribution counts equally and nobody can be excludedthe parts arrive from different stages and the fan-in fires once per key
money, safety, or policy turns on itload, backlog, counts, thresholds, readinessthe stage is data-driven rather than opinion-driven

Three decisions shape a vote. Granularity: one peer is one vote by default; name peer/agent in the vote grant and the count becomes one attested agent one vote, which is what lets a council of several agents sit on one peer. Quorum: too low and one peer decides, too high and a lost peer stalls the vote until its lease lapses; three of five is the usual answer, and an odd electorate avoids ties (a tie closes to the lexicographically first option, so avoid relying on it). Who opens it: a stage returns a Motion when its question, options, or quorum depend on the cue, or carries @Propose when the shape is fixed and only the key varies; either way the proposal opens once per id however many replicas see the cue.

Two decisions shape an estimate. What settles it: a settle rule fires when the estimate has held for N steps within a tolerance, a step being a gossip round the node took part in once it has heard the epoch, or the epoch going quiet, so a node nobody pushes to never settles on its own value and a plateau before a late contribution is not settled; pick N from the fleet's diameter and the spread of the contributions (example 18 needs forty at a 250 ms tick, since one reserve follows the ordered log) and the tolerance from what the consumer can tolerate. Who takes part: shares go only to members whose aggregate advertisement is live, so a console or a client that has not started the capability never swallows mass; a member that must count, must advertise. An estimate does not detect a poisoned contribution; the moment one would matter, the decision is a vote.

MAJORITY_GOSSIPThe vote capability's gossip mode decides by push-sum over preferences: fast, approximate, no ballots to audit. It is the right tool for a leader hint or a soft threshold and the wrong tool for anything the vote's audit trail exists for. The annotations bind QUORUM votes; the gossip mode is reached through the capability's helpers.
aspace://architecture/workflows · chapter 9

When a workflow, and how many spaces

A workflow on this fabric is nothing more than records that carry a case id moving through spaces, each stage a binding that takes or reacts and writes the next record. That is why there is no engine: the vocabulary of the workflow guide (sequence, map, fork, fan-out, join, gather, reduce, compose, filter, route, branch, decide, exactly once, deadline, loop, compensation, the human step, the model step, the planner) is each a way of binding, and the engine's state is the spaces themselves. Reach for it when a business case passes through several hands and each hand must survive a crash; keep a single take when one worker does the whole job.

ShapeWhenWhat to watch
One space, one record type per stagea short pipeline whose stages are all takes and all workers read the laneevery stage's lease fits its work; the lane's strategy fits every stage
Several spaces, one groupstages differ in strategy, admission, or retention; a join gathers parts from several lanes; one stage needs the ordered loga part may live in another space; an ORDERED join lives on the space that hosts the log; reactions fire on every replica, so what must happen once is a take, a fleet-wide join, a vote, or an idempotent write
Several groupsthe parties differ: a tenant's flow hands a commitment to a partner's, a conversation group reports to a mission groupa bridge peer in both groups, bound by group name; the bridged record is the contract; no shared ordered log across groups, so the bridge takes once on one side and writes idempotently on the other

Deadlines are leased reminders: a stage writes a reminder with the lease it is willing to wait, and a reaction on the lease's expiry escalates; the reaction fires within a tick of the lapse and nowhere else. Loops are stages that write their own cue again with a bounded counter. Compensation is a stage like any other, written when a later stage fails, taken by the agent that can undo. A planner or a model is a stage whose implementation is a model call; the take's lease is renewed on every model round-trip by the host libraries, so a long think does not lapse the task.

Where it is worked throughExample 18 is a nine-stage claims flow over five spaces and three peers, with one stage per concept and a test that reads every property from a space; the workflow guide walks it chapter by chapter. The four flagship applications (petclinic, code, audit, intake) are complete flows with their own guides.
aspace://architecture/recipes · chapter 10

Deployment recipes

Edge fleet3 to 15 listening peers on onenetwork; every peer a seed;LEASE_RACE spaces; mtlsHub and fielda few listening seeds, manydial-only peers; relay forpeer-to-peer; the hub holds theordered logMulti-sitea rendezvous per site, relaysacross; one group; ORDERED onlywhere a majority stays livePartner integrationa second group; a bridge peer inboth; content keys; A2A or MCP atthe edgePolyglot workersJava coordinator with Python andTypeScript workers dialing in onplain TCP (dev-local) or fronted byTLSHuman in the loopa console peer withcommand-and-control grants; votesfor approvals; deadlines as leasedreminders
Six deployment recipes. Each is a composition of the patterns in this guide: a topology, a set of boundaries, a rung on the ladder for each space, a way to decide, and an edge for the outside world. The chapters above say when each piece earns its cost.
RecipeTopology (ch. 1 to 3)Boundaries (ch. 4, 5)Guarantees (ch. 6 to 9)Edge
Edge fleet3 to 15 listening JVM peers on one network, every peer a seed, mtlsone group, one space per lanelease race everywhere; three workers per kind; a vote where a human would have signeda console peer
Hub and fielda few listening seeds in the hub, many dial-only field peers (laptops, boxes, Python and TypeScript clients), a relay for field-to-fieldone group; field lanes admitted by allowlistthe hub holds the ordered log; field work is lease-race takes with long leasesa console and, for partners, an A2A gateway in the hub
Multi-sitea rendezvous per site, relays across, one groupspaces per lane; a content key if the sites' links are not trustedordered log members on the site with the steadiest links; settle windows sized to the WANa gateway per site if partners dial in
Partner integrationthe partner runs peers of its own or dials an A2A gatewaytwo groups and a bridge peer; the partner group under POLICY or INVITE; a content key per groupcommitments on an ORDERED space in the group that owns them; idempotent writes across the bridgeA2A for agent protocols, MCP for model tooling
Polyglot workersa JVM coordinator with Python and TypeScript workers dialing inone group; the workers' lanes admitted by allowlist; dev-local or a TCP-fronting peerclients take under the lease race and contribute to estimates and votes; once-only stages stay on the JVMthe trilingual demo's council is the template
Human in the loopany of the above plus a console peer with the directive grantan audit space for every commandapprovals as votes with the panel in the grant; deadlines as leased reminders that escalatethe console's command-and-control: dispatch, cancel, pause, resume, drain
aspace://architecture/checklist · chapter 11

The architect's checklist

  • Peers. Each peer's purpose is one of worker host, coordinator, rendezvous, relay, console, gateway, or bridge, and the count of each follows chapter 1's table. Every worker kind has three or four instances on different peers.
  • Clock. The tick and gossip period are chosen, documented, and the same for every member; every lease, settle window, and deadline is a stated multiple of them.
  • Bootstrap. Every peer has at least two seeds; each site has a rendezvous; dial-only peers that must reach each other have a relay. The founding path (starter string, or a kept founder key in code) is decided with the revocation and rotation paths it implies.
  • Profiles. Every peer runs a security profile that meets the others'; where a non-JVM client dials in, the TCP listener it reaches is deliberate and require-attestation is off only where that is intended.
  • Lanes. Each space has one strategy, one admission rule, one retention, and a reason for existing; records carry the case id; references are ids.
  • Perimeters. Each group has a membership policy, a content-key decision, and an authorizer with named grants; any bridge peer is listed with the records it carries across.
  • Rungs. Each space's rung on the ladder is stated with the workload fact that justifies it; ordered logs have a named member set of three or five and the irreversible call is last.
  • Decisions. Each decision is a vote or an estimate with the reason; votes have a quorum, a granularity, and an electorate; estimates have a settle rule and only advertising participants.
  • Failure. For each stage: what lapses, who picks it up, what a reaction drops; for each partition: which side keeps working and which stops.
  • Observation. A console peer exists; the operator can name the written, completed, queued, and in-progress counts per space and the suspect state per peer; every command is audited.
  • Proof. The flow has a test that reads every property from a space, runs under a crash, and (where languages mix) runs the workers in their own processes.
Where to go nextThe developer guide for each API and property named here, the workflow guide for the stage patterns and example 18, RELIABILITY.md for the availability arithmetic, and SPEC.md and TECH-SPEC.md for the formats and the rationale.