The idea: a clinic that runs itself
A pet owner phones the clinic. A front desk writes the request down, a nurse decides how urgent it is, a scheduler books a vet, and the vet writes the chart. In most software that sequence has a conductor: one service that calls the nurse, takes the answer, calls the scheduler with it, and so on down the line. Pet Clinic has no conductor. The four people share one record-keeping space, and each watches for the one kind of record that is its cue.
The clinic is deliberately small. It has four record types, four kinds of agent, one space, and one rule for every agent: react to the entry you understand and write the entry the next stage is waiting for. That rule is service choreography, and this guide uses the clinic to show how it works, how the binder carries the hard parts for the author, and where it needs care.
Each agent is an ordinary object with one annotated method. The annotation names the space to watch, the method parameter names the entry type that is the cue, and the return value is the next entry. Nobody wires the graph: the nurse produces a Triage, the scheduler consumes a Triage, and the flow exists because those two types are the same.
Here is a whole agent, the nurse, in 9 lines. It reads a VisitRequest, rates the reason, and returns a Triage. It holds no queue, no thread, and no reference to the scheduler.
@AgentSpec(name = "triage-nurse", description = "Assigns acuity to each visit request", goals = {"triage"}) public static final class Nurse { @SpaceNotify(space = CLINIC, lease = "4h", resultLease = "4h") public Triage assess(VisitRequest request) { return new Triage(request.requestId(), request.petName(), triageAcuity(request.reason()), "triaged from: " + request.reason()); } }
| Workflow orchestration | Service choreography, as in this clinic | |
|---|---|---|
| Where the order lives | in one conductor that calls every stage | in the record types: a Triage is the scheduler's cue because the nurse writes one |
| What a stage knows about the others | their addresses and their APIs | nothing; it knows the type it watches and the type it writes |
| Adding a stage | edit the conductor | bind a new agent whose cue is a type that already exists, such as a billing agent watching VisitRecord |
| Where the work waits | in the conductor, or in a queue it owns | in the space, as entries with leases, until the next stage reacts |
You have run the quick start, which is a two-stage flow of this shape. This guide takes it to four stages on 6 peers in the test and 5 in the demo. The next chapters introduce the cast, the way the peers form one group, the space and its records, and one visit end to end. Three chapters then go one level down: what the binder does for the author of a @SpaceNotify method (chapter 6), how the scheduler keeps its slot book in the space (chapter 7), and why the clinic's space waits 500 ms before it believes a take (chapter 8).
The cast
Four kinds of seat make a clinic. The front desk is plain code that writes VisitRequest entries and reads VisitRecord charts; it has no agent. The other three are classes with an @AgentSpec and one cue annotation. The demo seats 5 peers and 4 agents. The test seats 6 peers and 5 agents, because it adds a second scheduler on its own peer so that two schedulers share one slot book (chapter 7).
| Agent | Peer | Annotation | Cue, then output |
|---|---|---|---|
| front desk, plain code | front-desk | space.write | writes VisitRequest; reads VisitRecord to print the charts |
triage-nurse, class Nurse | triage-nurse | @SpaceNotify | VisitRequest, then Triage |
scheduler, class Scheduler | scheduler, and scheduler-2 in the test | @SpaceReduce | Triage and the current SlotBook, then a new SlotBook and an Appointment |
vet, class Vet | dr-song and dr-ruiz | @SpaceNotify | Appointment, then a VisitRecord, or nothing for a pet booked with the other vet |
The two vets are one class with a constructor argument. dr-song is the on-call vet: the scheduler sends every urgent and emergency case there. Both vets receive every Appointment, and the guard on the first line of see ignores the ones addressed to the other vet. That guard is all the routing the clinic has.
The peers and the group
A peer is a process with a key, a node on a port, and a replica of every space the group holds. PetClinicFleet.startPeer builds one, and the demo and the test call it once per seat: the demo for front-desk, triage-nurse, scheduler, dr-song, dr-ruiz, the test for front-desk, triage-nurse, scheduler, scheduler-2, dr-song, dr-ruiz. Each call names a TCP port and a seed port. The front desk starts first with seed port 0, so it has nobody to dial, and every later peer lists the front desk as its seed.
The group is founded from a literal. group() hashes the text petclinic-fleet-v1, so every peer computes the same group id, aspace://zEfEgtojhKwEWzTtqr5f3Yfgkm7puv2JxhNXZAoA4VRCK, with no registry and no document to distribute. Each peer builds its own advertisement around that id with a throwaway issuer key, which changes nothing about the id. Because the founding is a text literal, the group has no signed founding document, so a peer joins it by dialing a seed address, as the figure shows.
public static GroupAdvertisement group() { GroupId groupId = GroupId.fromFounding( "petclinic-fleet-v1".getBytes(StandardCharsets.UTF_8)); return new GroupAdvertisement("aspace://" + groupId.value(), PeerIdentity.generate().peerId(), groupId, Instant.EPOCH, Duration.ofDays(365), "petclinic-fleet", GroupAdvertisement.MembershipPolicy.OPEN, ConflictStrategyType.LEASE_RACE, GroupAdvertisement.GossipParameters.defaults()); }
| What | Value | Where it comes from |
|---|---|---|
| group name | petclinic-fleet | group() |
| group id | aspace://zEfEgtojhKwEWzTtqr5f3Yfgkm7puv2JxhNXZAoA4VRCK | SHA-256 of the text petclinic-fleet-v1, base58btc |
| membership policy | OPEN: anyone who dials may join | GroupAdvertisement.MembershipPolicy |
| default strategy | LEASE_RACE | group(); the clinic space inherits it |
| gossip | fan-out 3, period 1 s | GossipParameters.defaults() |
| membership | member TTL 30 s, ping timeout 2 s, 2 indirect probes | GroupMembership.Config in startPeer |
| transport | TcpTransport on 127.0.0.1; the demo uses ports 7781 to 7785, the test asks the operating system for a free port per peer | startPeer, freePort() in the test |
| security | frames signed with each peer's Ed25519 key; no TLS, no profile, no keys to configure | PeerNode.builder(identity).build() uses the default ChannelAuth.SIGNED |
| tick | 250 ms | node.startTicking |
PeerIdentity identity = PeerIdentity.generate(); PeerNode node = PeerNode.builder(identity).build(); node.listen(new TcpTransport(), HOST + ":" + port); List<PeerAdvertisement.Endpoint> seeds = seedPort == 0 ? List.of() : List.of(new PeerAdvertisement.Endpoint("tcp", HOST + ":" + seedPort, 0)); GroupRuntime runtime = node.joinGroup(group(), new GroupMembership.Config(Duration.ofSeconds(30), Duration.ofSeconds(2), 2), seeds);
// The settle window is how long a take waits for a competing claim to arrive; // it has to outlast the time a claim needs to reach the other peers. The // library's default is 200 ms. Six peers on one loaded machine push a claim in // well under 500 ms; at 150 ms the two schedulers sometimes both won the race // for one triage and booked it twice. ReplicatedSpace clinic = ReplicatedSpace.builder(runtime, CLINIC, identity, name) .settleWindow(Duration.ofMillis(500)) .build();
// The agents are annotated POJOs bound through the facade (the starter // does this for Spring beans); no identity or threading code in them. The // discovery service is null on purpose: this teaching fleet routes by entry // type and publishes no agent cards. AgentSpaces spaces = new AgentSpaces(identity, InstantSource.system()); AgentSpaces.GroupContext group = spaces.register("petclinic-fleet", runtime.id(), runtime, null); group.space(CLINIC, clinic); node.startTicking(Duration.ofMillis(250));
Everything the agents share is the clinic space that every peer builds in startPeer. The front desk keeps the ReplicatedSpace handle and reads and writes it directly; the other peers hand it to the facade, so the binder can find it by name. The agents are bound after the peer is up, with runNurse, runScheduler, and runVet, each of which is one call to group().bind(...).
public static AutoCloseable runNurse(Peer peer) { return peer.group().bind(new Nurse()); } public static AutoCloseable runScheduler(Peer peer) { return peer.group().bind(new Scheduler()); }
<agentspaces.version>0.5.0</agentspaces.version> <dependency> <groupId>ai.badmonkey.agentspaces</groupId> <artifactId>agentspaces-agent</artifactId> <version>${agentspaces.version}</version> </dependency>
How the agents find each other
The clinic's agents do not look for each other. Each agent looks for an entry type, and the space brings the entries. When the binder binds the nurse it reads the method's parameter type, VisitRequest, builds a template from it, and subscribes to the clinic space. Because the space replicates, a VisitRequest written on the front desk's peer appears on the nurse's peer a moment later and fires that subscription. The same holds for every other hop.
| Entry type | Who watches it | How | What happens |
|---|---|---|---|
VisitRequest | the nurse | @SpaceNotify on the parameter type | writes a Triage |
Triage | the scheduler | @SpaceReduce on the element type | takes it and writes a SlotBook and an Appointment |
Appointment | both vets | @SpaceNotify plus a guard on vet() | the addressed vet writes a VisitRecord, the other returns null |
VisitRecord | nobody | the front desk reads it with readAll | the chart is the end of the flow |
The peers find each other once, at startup. A newcomer dials the seed, learns the members from it, and from then on gossip carries membership, advertisements, and the spaces' entries, as the join figure of chapter 2 shows. The demo waits 1500 ms after starting the five peers before it writes the first request, which gives the mesh time to form; the test waits the same.
Discovery through cards is the other way a fleet finds its services, and this app publishes none, on purpose. startPeer registers the group with a null discovery service, and the binder publishes an agent card only when it has one to publish to, so no card exists in this fleet. Code Fleet and Intake Fleet, the other two teaching fleets, make the same choice, because the entry types route the work and nothing in them reads a card. If you pass a DiscoveryService, the binder publishes one card per bound agent, each listing what the agent consumes and produces: triage-nurse consumes VisitRequest and produces Triage, scheduler consumes Triage and produces SlotBook and Appointment, and vet consumes Appointment and produces VisitRecord. A console or a planner could then list the clinic's services, as the figure shows with dashed boxes for the cards that do not exist here.
The clinic space
The clinic has one space, clinic, a replica on every peer. It uses the LEASE_RACE strategy: a take is decided by a claim race that waits one settle window (500 ms here, chapter 8). Five record types move through it. The Triage is consumed by a take, and only the scheduler writes and reads the SlotBook, so it is state rather than a message.
| Record | Written by | Read by | Lease | How it ends |
|---|---|---|---|---|
VisitRequest | the front desk | the nurse | 4 hours, set by the front desk | the lease lapses |
Triage | the nurse | the scheduler, which takes it | 4 hours, the nurse's result lease | the scheduler completes it with the new SlotBook |
SlotBook | the scheduler | the scheduler, and any peer through Reductions.current | 7 days, the reduce's accumulator lease | the next fold replaces it |
Appointment | the scheduler, after the book | both vets | 7 days, the reduce's accumulator lease | the lease lapses |
VisitRecord | the addressed vet | the front desk | 4 hours, the vet's result lease | the lease lapses |
public record VisitRequest(String requestId, String petName, String owner, String species, String reason) { }
public record Triage(String requestId, String petName, String acuity, String note) { }
public record SlotBook(int routineTurn, int nextSlot) { }
public record Appointment(String requestId, String petName, String vet, String slot, String acuity) { }
public record VisitRecord(String requestId, String petName, String vet, String diagnosis, String treatment) { }
JoinTicket entry per key to serialize its folds (chapter 7). It is the only entry type in the space the clinic's code does not declare.One visit, end to end
The demo's front desk takes four requests, writes them, and walks away. Everything after the write is a reaction. The figure follows Rocky, the emergency, through the five peers of the demo: the front desk writes a request, the nurse rates it and writes a triage, the scheduler folds it into the slot book and writes an appointment, both vets receive the appointment, and the vet it names writes the chart.
Step 1: the front desk writes
The demo files four requests. Nothing in them names an acuity or a vet: the owner gives a reason in free text, and the rest of the clinic works it out.
List<VisitRequest> requests = List.of( new VisitRequest("v-1", "Bella", "Ada", "dog", "overdue for annual checkup"), new VisitRequest("v-2", "Milo", "Ben", "cat", "not eating, lethargic 2 days"), new VisitRequest("v-3", "Rocky", "Cleo", "dog", "hit by car"), new VisitRequest("v-4", "Luna", "Dov", "rabbit", "routine vaccination booster"));
for (VisitRequest request : requests) { frontDesk.clinic().write(request, Lease.of(Duration.ofHours(4))); }
Step 2: the nurse rates the reason
The nurse's method runs once for each VisitRequest on the nurse's peer. The rule is a keyword check that stands in for a real triage protocol: it reads the reason and returns one of three acuity levels.
@AgentSpec(name = "triage-nurse", description = "Assigns acuity to each visit request", goals = {"triage"}) public static final class Nurse { @SpaceNotify(space = CLINIC, lease = "4h", resultLease = "4h") public Triage assess(VisitRequest request) { return new Triage(request.requestId(), request.petName(), triageAcuity(request.reason()), "triaged from: " + request.reason()); } }
public static String triageAcuity(String reason) { String r = reason.toLowerCase(); if (containsAny(r, "hit by car", "bleeding", "seizure", "collapse", "not breathing", "poison")) { return "EMERGENCY"; } if (containsAny(r, "not eating", "lethargic", "vomiting", "limping", "in pain", "swelling")) { return "URGENT"; } return "ROUTINE"; }
@SpaceNotify, lease = "4h" is the lifetime of the subscription, which the binder renews at half the lease while the agent stays bound, and resultLease = "4h" is the lifetime of the entry the method returns.Step 3: the scheduler books
The scheduler takes each Triage, folds it into the slot book, and returns two entries: the new book and the Appointment. Urgent and emergency cases go to dr-song with the slot immediate. Routine cases alternate between the vets and take the next half hour, counted from 09:00 (chapter 7 tells how the book stays consistent).
@AgentSpec(name = "scheduler", description = "Books each triaged visit with a vet", goals = {"schedule"}) public static final class Scheduler { @SpaceReduce(space = CLINIC, name = "slot-book", lease = "30s", accumulatorLease = "7d", produces = Appointment.class) public Entries book(SlotBook book, Triage triage) { SlotBook current = book == null ? new SlotBook(0, 0) : book; boolean urgent = triage.acuity().equals("EMERGENCY") || triage.acuity().equals("URGENT"); String vet = urgent ? ON_CALL : VETS.get(current.routineTurn() % VETS.size()); String slot = urgent ? "immediate" : slotLabel(current.nextSlot()); SlotBook next = urgent ? current : new SlotBook(current.routineTurn() + 1, current.nextSlot() + 1); return Entries.of(next, new Appointment(triage.requestId(), triage.petName(), vet, slot, triage.acuity())); } }
private static String slotLabel(int index) { int minutes = 9 * 60 + index * 30; // slots from 09:00 in half-hours return String.format("%02d:%02d", minutes / 60, minutes % 60); }
Step 4: the vets answer
Both vets receive every Appointment, because a notify subscription reports each entry to every agent bound on every peer. The vet the appointment names diagnoses the pet and returns a VisitRecord. The other vet returns null, and the binder writes nothing for a null.
@AgentSpec(name = "vet", description = "Sees the pets booked with it and writes the chart", goals = {"treat"}) public static final class Vet { private final String vetName; public Vet(String vetName) { this.vetName = vetName; } @SpaceNotify(space = CLINIC, lease = "4h", resultLease = "4h") public VisitRecord see(Appointment appointment) { if (!appointment.vet().equals(vetName)) { return null; // not mine } String[] outcome = diagnose(appointment.petName()); return new VisitRecord(appointment.requestId(), appointment.petName(), vetName, outcome[0], outcome[1]); } }
Step 5: the charts
The front desk polls its own replica for VisitRecord entries until all four have arrived, then prints them. The loop is the only place the clinic waits for anything, and it belongs to the demo: the agents never wait.
for (int tick = 0; tick < 40; tick++) { records = frontDesk.clinic().readAll(Template.of(VisitRecord.class), 100); if (records.size() >= requests.size()) { break; } sleep(500); }
The four pets, as the demo ran them
The acuity column is what the keyword rule in triageAcuity gives for each reason; the build computes it from the source and checks it against the recording. The vet and slot columns are from one recorded run (2026-11-04). The emergency and the urgent case always go to dr-song at once. The two routine pets split between the vets and take 09:00 and 09:30, but which pet gets which vet depends on the order the scheduler folds them, so it can change from run to run.
| Request | Pet | Reason | Acuity | Booked |
|---|---|---|---|---|
v-1 | Bella, a dog, owner Ada | “overdue for annual checkup” | ROUTINE | dr-song at 09:00 |
v-2 | Milo, a cat, owner Ben | “not eating, lethargic 2 days” | URGENT | dr-song at immediate |
v-3 | Rocky, a dog, owner Cleo | “hit by car” | EMERGENCY | dr-song at immediate |
v-4 | Luna, a rabbit, owner Dov | “routine vaccination booster” | ROUTINE | dr-ruiz at 09:30 |
petclinic-fleet: a clinic that runs itself by choreography Nov 04, 2026 1:43:13 AM ai.badmonkey.agentspaces.agent.reduce.ReduceBinding start INFO: slot-book: LEASED reduce of Triage into SlotBook on 'clinic', keyed by nothing (one accumulator), 0 key(s) pending front desk logged 4 visit requests; the clinic takes it from here == the day's charts == Bella dr-song 09:00 @ ROUTINE [seen] dx: healthy on annual exam, slightly overweight rx: diet plan and dental cleaning, recheck in six months Milo dr-song immediate @ URGENT [seen] dx: early gastrointestinal infection rx: subcutaneous fluids and antibiotics, recheck in 48h Rocky dr-song immediate @ EMERGENCY [seen] dx: stable after collision; no fractures on X-ray rx: overnight observation, pain management, recheck in 24h Luna dr-ruiz 09:30 @ ROUTINE [seen] dx: healthy; due vaccinations administered rx: booster given, no follow-up needed Every pet was requested, triaged, scheduled, and seen with no coordinator: the nurse watched for requests, the scheduler for triage, the vets for their own appointments. The space carried the work from one to the next.
What the binder does for the author
The nurse's method is short because the binder carries the discipline that every choreography agent needs. A notify subscription delivers events at least once, on the fabric's thread, and a slow method on that thread would stall every other delivery. The binder gives the author three guarantees, and the method body states none of them.
- Each entry reaches the method once. The binder keeps a set of entry ids per subscription, bounded at 4096 ids, and drops a delivery whose id it has seen. At-least-once delivery becomes once per bound agent.
- The reaction runs off the delivery thread. Each delivery starts a virtual thread named after the class and method, so a slow method never holds up the fabric.
- The binder writes the return value back. A non-null return becomes the next entry, which the binder writes to the result space (the watched space by default) under
resultLease;nullwrites nothing.
private Subscription subscribe(Object agent, Method method, SpaceNotify notify, Class<?> entryType, Bound handle) { Space space = spaceFor(handle.identity, notify.space(), method); Space resultSpace = notify.resultSpace().isEmpty() ? space : spaceFor(handle.identity, notify.resultSpace(), method); Lease lease = Lease.of(Durations.parse(notify.lease())); Lease resultLease = Lease.of(Durations.parse(notify.resultLease())); // Delivery is at-least-once; the binder carries the choreography // discipline so the method body never has to: dedupe per entry id (watch), // then run the reaction on its own virtual thread so slow work never // holds the fabric's delivery thread, and write a non-null result back // as the next entry in the flow. String label = agent.getClass().getSimpleName() + "." + method.getName(); Template<?> template = templateFor(entryType, notify.tags(), notify.where(), method); if (notify.on().length == 0) { throw new IllegalArgumentException("@SpaceNotify at " + method + " reacts to no event kind"); } return watch(space, template, lease, handle, "space-notify", label, Set.of(notify.on()), spaceEvent -> Thread.ofVirtual().name("space-notify-" + label).start(() -> { try { Object result = invoke(agent, method, spaceEvent.entry()); dispatch(result, null, resultSpace, resultLease, null, handle, method, label); } catch (RuntimeException e) { LOG.log(System.Logger.Level.WARNING, label + ": notify failed", e); } })); }
Set<Object> seen = java.util.Collections.newSetFromMap( new java.util.LinkedHashMap<>() { @Override protected boolean removeEldestEntry(Map.Entry<Object, Boolean> eldest) { return size() > NOTIFY_DEDUP_CAPACITY; } }); Subscription subscription = space.notify((Template) template, event -> { SpaceEvent<?> spaceEvent = (SpaceEvent<?>) event; if (!kinds.contains(spaceEvent.kind())) { return; } // Once per entry per kind: a WRITTEN and a later EXPIRED of one entry are // two deliveries; a redelivered WRITTEN is not. Object key = kinds.size() == 1 ? spaceEvent.entryId() : spaceEvent.entryId().value() + ":" + spaceEvent.kind(); synchronized (seen) { if (!seen.add(key)) { return; // redelivery of an event this agent already handled } } onEvent.accept(spaceEvent); }, lease);
private void dispatch(Object result, Space space, Space resultSpace, Lease resultLease, TakenEntry<?> taken, Bound handle, Method site, String label) { if (result == null) { if (taken != null) { space.complete(taken); } return; }
if (tags.isEmpty()) { resultSpace.write(entry, resultLease); } else { resultSpace.write(entry, resultLease, tags); }
The same subscription, bound on two peers, runs twice: each peer has its own binder with its own set of seen ids. That is the right behavior for the vets, who both need to see each Appointment, and the wrong behavior for work that exactly one agent should do. A second nurse would write a second Triage for every request. Work that one agent must do goes through a take, which is why the scheduler uses @SpaceReduce and why the test can bind it twice.
What to watch for
- A subscription starts now. The binder registers it when the agent binds, and it sees entries that arrive afterward. An entry already in the agent's replica when the agent binds, such as a request written while the agent was down, never fires the method (chapter 9).
- The set of seen ids lives in memory. A restarted agent starts with an empty set.
- Reactions can overlap. Each delivery gets its own thread, so two requests that arrive together run concurrently and finish in either order. The clinic's methods are pure, so this costs nothing; an agent that shares state needs its own locking.
- A method that throws loses that delivery. The binder logs a warning and writes nothing, and the id is already in the seen set, so it does not retry. A take handles failure differently: the lease lapses and the entry comes back.
@SpaceNotify and the publish and subscribe table of the workflow guide maps the broker terms onto it. The javadoc is SpaceNotify.The slot book: state in the space
The scheduler has the one job in the clinic that needs memory: whose turn it is for the next routine visit, and which half hour is next. The memory has to survive a restart, and in the test it has to be shared by two schedulers on two peers. So the book is not a field on the scheduler. It is a SlotBook entry in the clinic space, advanced once per triage by @SpaceReduce: a persistent accumulator whose state lives beside the entries it folds.
@SpaceReduce(space = CLINIC, name = "slot-book", lease = "30s", accumulatorLease = "7d", produces = Appointment.class)
The method takes the current book and one Triage, and returns an Entries fork of the new book and the Appointment. The binder treats the first element, the one of the accumulator type, as the new book and the rest as side outputs. It holds a ticket for the key while it folds. With no key on the annotation every Triage folds into one accumulator, which serializes all the clinic's bookings through one ticket.
if (!tickets.request(key)) { return; } Optional<TakenEntry<JoinTicket>> ticket = take(tickets.template(key)); if (ticket.isEmpty()) { return; // another reducer holds the key; its events or the sweep raise it again }
long stepCount = current.map(Reductions::steps).orElse(0L) + 1; tags.put(Reductions.REDUCE_TAG, config.name()); tags.put(Reductions.KEY_TAG, key); tags.put(Reductions.STEPS_TAG, Long.toString(stepCount)); EntryHandle handle = config.space().complete(element, accumulator, config.accumulatorLease(), tags); EntryHandle previous; synchronized (lock) { previous = written.put(key, handle); } current.ifPresent(predecessor -> retire(key, predecessor, previous)); for (Object other : others) { sideOutputs.accept(other); }
The step is atomic: completing the Triage with the new book is one operation, so one reducer never folds a Triage twice and the Triage is never completed without its new book. The binder writes the Appointment after that step, as the fork's side output.
// A scheduler restarts and a fifth routine visit lands: the book, not the // scheduler's memory, says the next slot is 10:00. firstScheduler.close(); subs.add(PetClinicFleet.runScheduler(scheduler)); frontDesk.clinic().write(new VisitRequest("v-5", "Pip", "Eve", "cat", "routine dental check"), Lease.of(Duration.ofHours(4))); List<VisitRecord> five = awaitRecords(frontDesk, 5, Duration.ofSeconds(60)); assertThat(five).extracting(VisitRecord::petName).contains("Pip"); Appointment pip = frontDesk.clinic().readAll(Template.of(Appointment.class), 100).stream() .filter(a -> a.petName().equals("Pip")).findFirst().orElseThrow(); assertThat(pip.slot()).isEqualTo("10:00"); assertThat(Reductions.current(frontDesk.clinic(), SlotBook.class, "slot-book", Reductions.GLOBAL_KEY).orElseThrow().value()).isEqualTo(new SlotBook(3, 3));
// The slot book is an entry: two routine visits booked, two slots gone. The charts // arrive from the vets' peers before the scheduler's last fold has replicated to the // front desk, so the accumulator is awaited like the records. SlotBook book = awaitSlotBook(frontDesk, new SlotBook(2, 2), Duration.ofSeconds(30)); assertThat(book).isEqualTo(new SlotBook(2, 2)); assertThat(List.of(appts.get("Bella").slot(), appts.get("Luna").slot())) .containsExactlyInAnyOrder("09:00", "09:30");
LEASED mode a reduce takes its ticket through the space's claim race. Two reducers folding one key at once is therefore possible when a competing claim arrives late, which is the case chapter 8 explains. A fleet that needs exactly one reducer per key, fleet-wide, uses Mode.ORDERED, which sends the ticket and the takes through the ordered log. Pet Clinic stays on LEASED and a long settle window.The settle window
A LEASE_RACE take is a claim race. A taker writes a claim on the entry, publishes it to the group, and waits one settle window before it decides. After the window it keeps the entry only if its own claim is the lowest one it knows: the claim with the lowest (hlc, peerId) pair wins. The window exists so that a rival's claim has time to reach the taker before the taker decides.
publishDelta(new SpaceWire.Delta(null, claimedId, sign(myClaim, actor)), "c:" + claimedId + ":" + myClaim.stamp().encoded()); sleep(settleWindow); synchronized (lock) { TakeClaim winner = claimOf(claimedId); boolean won = winner != null && winner.holder().equals(holder) && winner.epoch() == myClaim.epoch() && !winner.expired(nowMillis()); if (!won) { return Optional.empty(); // lost the race; retry elsewhere } takenEpochs.put(claimedId, myClaim.epoch()); }
A rival claim that arrives after the window leaves two takers who both believe they hold the entry. The SPEC allows that (SPEC.md section 7.4: transient duplicate work remains possible), and it is what happened to this app's test. The changelog entry of 2026-11-02 records that PetClinicFleetTest was flaky at 0.5.0, passing one run in four. A diagnostic run showed one Triage folded by both schedulers, each writing an Appointment for it. The clinic space then waited 150 ms with six peers on one loaded machine. It now waits 500 ms, and eight runs in a row passed after the change.
// The settle window is how long a take waits for a competing claim to arrive; // it has to outlast the time a claim needs to reach the other peers. The // library's default is 200 ms. Six peers on one loaded machine push a claim in // well under 500 ms; at 150 ms the two schedulers sometimes both won the race // for one triage and booked it twice. ReplicatedSpace clinic = ReplicatedSpace.builder(runtime, CLINIC, identity, name) .settleWindow(Duration.ofMillis(500)) .build();
The library's own default is a fixed 200 ms. The builder and the Spring starter agree on it, and it does not follow the gossip period. The clinic overrides it: the code in the repository today waits 500 ms, and its comment says why. On 2026-11-04, three runs of the test passed against the build installed from the workspace, and three more against the 0.5.0 jars from Maven Central (chapter 10 has the output). The number is a trade. Every take waits the full window, so the scheduler, the only agent that takes, spends at least 500 ms per take; the nurse and the vets react without a claim and pay nothing. A longer window makes a late claim less likely, and it does not make one impossible.
What to watch for
- Size the window for the slowest claim. The window has to outlast the time a claim needs to cross the group on a loaded machine. The test runs six peers in one JVM, which is why a window that suits a quiet fleet failed here.
- The demo does not race. It binds one scheduler, so no rival claim exists. The test binds two to put the race under test.
- Make the work under a take idempotent. The SPEC advises it for every
LEASE_RACEworkload, because the window narrows the duplicate case and cannot close it. - Use the ordered log when one is required.
Mode.ORDEREDon the reduce decides the ticket by log order, at the cost of a Raft quorum (chapter 7).
When something stops
Three things can stop in the clinic, and the space handles each differently. The difference is the most useful thing to take from a choreography: work that a take holds comes back when its lease lapses, state that lives in an entry outlives the agent that wrote it, and a reaction that no agent was bound to receive is not repaired.
A scheduler dies or its fold throws
The reduce takes each Triage under a 30s lease. If the method throws, the binder logs it and stops that drain; if the peer dies, nothing completes the take. Either way the lease lapses, the Triage reappears, and a scheduler takes it again. The code is the same as for the happy path, because the crash path is the absence of a completion. This app's test does not kill a scheduler in the middle of a fold, so the behavior comes from the binder's design and its own tests.
Object result; try { result = step.apply(current.map(Space.Entry::value).orElse(null), element.entry()); } catch (RuntimeException | Error e) { LOG.log(System.Logger.Level.WARNING, config.name() + ": the fold threw for key '" + key + "'; the element's take lease will lapse", e); return false; }
A scheduler restarts
The test closes the first scheduler's binding and binds a new one on the same peer. The slot book is an entry with a lease of 7 days, so it is still in the space. The new binding sweeps the space for waiting elements when it starts, reads the book, and books the fifth pet at 10:00. The test excerpt in chapter 7 shows it; its last assertion is the book itself, SlotBook(3, 3).
The nurse is down when a request arrives
This is the case the space does not repair. A @SpaceNotify subscription starts when the agent binds and delivers only the entries that arrive after it. A VisitRequest written while the nurse is down stays in the space for its 4 h lease. A nurse bound again on the same peer finds the request already in its replica, so no event fires and no Triage follows. A nurse that starts as a new process begins with an empty replica and sees the request fire only when a sync from another peer delivers it after the nurse has bound, a race the clinic does not rely on. A reduce differs: it sweeps the space when it starts, so a scheduler that binds late still folds a Triage that was already waiting. The workflow guide's publish and subscribe table says the same: the entry is the durable thing, and an agent that restarts reads the space.
VisitRequest entries when it binds, or the request stage could be a @SpaceTake so that an unanswered request reappears for the next nurse. The first keeps the nurse a reaction; the second turns the stage into a worker.Running and testing it
The app needs JDK 21 and Maven. It uses no model, no network service, and no API key: the diagnoses come from a canned table, and the peers talk over loopback TCP. The AgentSpaces 0.5.0 libraries resolve from Maven Central.
# in agentspaces-petclinic-fleet mvn test # the clinic as a flow test over real TCP mvn -q exec:java # the live demo: four pets, five peers, no orchestrator ./run-demo.sh # the same demo, compiling first
The demo binds ports 7781 to 7785 on 127.0.0.1, so stop one run before you start another. The test asks the operating system for a free port per peer and has no such limit.
What the test asserts
PetClinicFleetTest is one test of 6 peers (front-desk, triage-nurse, scheduler, scheduler-2, dr-song, dr-ruiz) running 5 agents. It writes the four requests and a fifth after a scheduler restart, and it asserts the following, in order.
- every pet reaches a chart with a diagnosis and a treatment, within 60 seconds;
- the acuity on each appointment follows the reason: Rocky
EMERGENCY, MiloURGENT, Bella and LunaROUTINE; - no
Triageis left in the space, so every one was taken and booked; - Rocky and Milo are booked with
dr-songatimmediate; - Bella and Luna are booked with the two different vets, at 09:00 and 09:30 in either order;
- each chart names the vet its appointment named;
- the slot book reads
SlotBook(2, 2), awaited because the last fold reaches the front desk after the charts do; - after the first scheduler closes and a new one binds, a fifth pet, Pip, is booked at 10:00 and the book reads
SlotBook(3, 3).
// The triage nurse assessed acuity from the reason for each visit; the // appointment carries it, since the scheduler's reduce consumed the triage. assertThat(appts.get("Rocky").acuity()).isEqualTo("EMERGENCY"); assertThat(appts.get("Milo").acuity()).isEqualTo("URGENT"); assertThat(appts.get("Bella").acuity()).isEqualTo("ROUTINE"); assertThat(appts.get("Luna").acuity()).isEqualTo("ROUTINE"); assertThat(frontDesk.clinic().readAll(Template.of(Triage.class), 100)) .as("every triage was taken and booked").isEmpty(); // The emergency and the urgent case both went to the on-call vet, at once. assertThat(appts.get("Rocky").vet()).isEqualTo(PetClinicFleet.ON_CALL); assertThat(appts.get("Rocky").slot()).isEqualTo("immediate"); assertThat(appts.get("Milo").vet()).isEqualTo(PetClinicFleet.ON_CALL); assertThat(appts.get("Milo").slot()).isEqualTo("immediate");
// The two routine visits were shared between the two different vets. assertThat(appts.get("Bella").vet()) .isNotEqualTo(appts.get("Luna").vet()); assertThat(List.of(appts.get("Bella").vet(), appts.get("Luna").vet())) .containsExactlyInAnyOrderElementsOf(PetClinicFleet.VETS);
int seed = freePort(); Peer frontDesk = PetClinicFleet.startPeer("front-desk", seed, 0); Peer nurse = PetClinicFleet.startPeer("triage-nurse", freePort(), seed); Peer scheduler = PetClinicFleet.startPeer("scheduler", freePort(), seed); Peer backup = PetClinicFleet.startPeer("scheduler-2", freePort(), seed); Peer song = PetClinicFleet.startPeer("dr-song", freePort(), seed); Peer ruiz = PetClinicFleet.startPeer("dr-ruiz", freePort(), seed); List<Peer> clinic = List.of(frontDesk, nurse, scheduler, backup, song, ruiz); // Two schedulers, on two peers, book from one slot book in the space. AutoCloseable firstScheduler = PetClinicFleet.runScheduler(scheduler); List<AutoCloseable> subs = new java.util.ArrayList<>(List.of( PetClinicFleet.runNurse(nurse), firstScheduler, PetClinicFleet.runScheduler(backup), PetClinicFleet.runVet(song, "dr-song"), PetClinicFleet.runVet(ruiz, "dr-ruiz")));
private static SlotBook awaitSlotBook(Peer peer, SlotBook expected, Duration timeout) throws InterruptedException { long deadline = System.nanoTime() + timeout.toNanos(); SlotBook book = null; while (System.nanoTime() < deadline) { book = Reductions.current(peer.clinic(), SlotBook.class, "slot-book", Reductions.GLOBAL_KEY) .map(e -> e.value()).orElse(null); if (expected.equals(book)) { return book; } Thread.sleep(250); } return book; }
mvn test, build of 0.5.0 installed from the workspace (the default local repository): run 1: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.840 s run 2: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.340 s run 3: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.312 s mvn -Dmaven.repo.local=<empty directory> test, the 0.5.0 jars from Maven Central: run 1: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.882 s run 2: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.943 s run 3: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.363 s
target/ can also leave stale test classes; mvn clean test rebuilds them.Where to go next
- Read the quick start again with the clinic in mind: it is the two-stage version of the same flow.
- The annotations the clinic uses are
@SpaceNotifyand@SpaceReduce; conflict strategies coversLEASE_RACE, the settle window, and the ordered log. - The workflow guide names the shapes this guide uses: publish and subscribe, the three reduces, and the exactly-once ladder.
- The Party Bus guide puts the same choreography beside an auction, a quorum vote, and an ordered log, and the Superfriends guide does it across three frameworks.
- Extend the clinic: add a billing agent that watches
VisitRecordand writes anInvoice. It is one class with one annotated method, and no existing agent changes.
agentspaces-petclinic-fleet in the example apps repository. The binder and the reduce this guide slices from are in agentspaces-agent, and the claim race is in agentspaces-space, both in the AgentSpaces repository.