The idea: an operations center that runs itself
Ops Center is the Autonomous Operations Center flagship of the AgentSpaces pattern catalog, written as a standalone Java project. An incident lands on a shared board. Specialist agents claim the parts they understand and write what they learn beside the incident. One triage worker dies in the middle of its job and the incident comes back to the other. Three judges, each on its own peer, disagree about the proposed remediation and a quorum decides. A person approves it from the fleet console, and only if the quorum approved it: a remediation the quorum rejected closes as rejected and never reaches an executor. The irreversible step runs once, because three executors race for the order through a Raft log. The closed incident carries every piece of evidence, and each line names the peer that signed it.
No agent calls another and nothing coordinates them from the middle. Each agent reacts to a record it understands in one of four working spaces and writes the next record, so the whole lifecycle is a chain of typed entries. Read this guide beside the developer guide, which explains each primitive, and the workflow guide, which names the patterns; this page shows one application that uses six of them together.
The scenario has two incidents. INC-1 is the hard one: the disk on db-primary fills, the first triage worker dies on the job, the proposed fail-over is irreversible, and the change-control judge dissents. INC-2 is the routine path: checkout latency, a reversible rollback, a unanimous panel. Both end with exactly one receipt. The demo, the test and this guide all run both. A third scripted incident exists for the other outcome. payments-vault reports a service account key exposed in a public repository, the proposed purge is irreversible and touches a credential, so two judges reject it and the quorum rejects it. The console session and the second fleet test raise it, and it closes as rejected with no approval and no receipt.
| The story beat | What carries it | Where in the code |
|---|---|---|
| an incident lands | a typed record in the incidents space; a monitor, a person or a planner can write one | OpsCenter.raise, the console's "Raise incident" |
| specialists claim parts | @SpaceTake under a 15 s lease for triage, @SpaceNotify reactions for the rest | Triage, Assets, Remediation |
| a worker dies and its task returns | the take lease lapses and the entry reappears | OpsCenter.CrashSwitch |
| three judges disagree and a quorum decides | @Propose, @Ballot, @OnDecision on the vote capability, one voter per peer | RiskPanel |
| a person approves what the panel approved | an OperatorApproval from the console or the simulated officer; the console and the dispatcher both refuse one for a rejected remediation | Watchtower, Operations.WatchOfficer, RiskPanel.refusal |
| a rejected remediation closes as rejected | the recorder's second cue, the RemediationDecision, writes an IncidentRecord with outcome rejected | Operations.Recorder.closeRejected |
| the irreversible step runs once | @OrderedTake over a three-member Raft log | Operations.Executor, OpsFleet.formExecutionQuorum |
| every piece of evidence, each with its signer | readAllIssued returns each entry with its issuer | Operations.Recorder |
| an operator watches the stages | the console's Incidents panel, one row per incident | Watchtower |
| a planner reaches the fleet | RemoteActions and the Embabel bridge turn cards into typed actions | Bridge, OpsCenterEmbabelApplication |
It runs on a laptop with no API keys. Without OPENAI_API_KEY the agents' models answer from a script, through the same Spring AI ChatClient path as a live model, format instructions and structured output included. With the key set, the same clients call OpenAI. The one profile that needs more is -Pembabel, which adds the Embabel planner.
The cast: twelve peers and fourteen agents
The center has 12 peers and 14 agents. Nine agent classes make the fourteen, because Triage, Judge and Executor run on several peers each. Every peer is a separate PeerNode with its own key, port and replicas, so closing one is what the fleet sees when a process dies. Two peers sit apart from the rest: the console hosts the human's seat, and the planner hosts no agent at all.
PeerNode with its own key, its own TCP port and its own replica of every space, and its second line names the agents it hosts. The planner hosts none: it reads the fleet's cards. Closing a peer is the event the rest of the fleet sees when a process dies.| Agent class (card name) | Peer | Annotation | Reads, then writes |
|---|---|---|---|
Triage (triage) | triage-a, triage-b | @SpaceTake on incidents, lease 15 s | Incident (and the Telemetry on record), then Diagnosis in evidence |
Assets (assets) | staff | @SpaceNotify on incidents | Incident, then Telemetry in evidence |
Remediation (remediation) | staff | @SpaceNotify on evidence | Diagnosis, then RemediationProposal |
Operations.Dispatcher (dispatcher) | staff | @SpaceNotify on evidence | OperatorApproval, the panel's RemediationDecision and the proposal, then RemediationOrder in actions; no order when the panel did not approve |
Operations.Recorder (recorder) | staff | 2 x @SpaceNotify on evidence | RemediationReceipt (or a rejecting RemediationDecision) and every entry about the incident, then IncidentRecord, outcome remediated or rejected |
RiskPanel.Lead (risk-lead) | risk-lead | @Propose, @OnDecision | RemediationProposal, then a vote Proposal; the closed vote, then RemediationDecision |
RiskPanel.Judge (risk-judge) | judge-security, judge-change, judge-ops | @Ballot on reviews | the vote Proposal, then a signed ballot and a RiskReview |
Operations.Executor (executor) | exec-a, exec-b, exec-c | @OrderedTake on actions, lease 30 s | RemediationOrder, then RemediationReceipt in evidence |
Operations.WatchOfficer (watch-officer) | console | @SpaceNotify on evidence | RemediationDecision, then OperatorApproval (headless only, and only for approve) |
Every @SpaceTake, @SpaceNotify, @OrderedTake and @Propose method also sets a description, which the card publishes as that action's description, so a planner that lists the fleet's actions reads what each one does. Two components are not agents. Watchtower is the fleet console, served from the console peer on port 7591, and Bridge gives the planner peer its view of the fleet. The method that boards all of this is OpsCenter.board: it starts the twelve peers in order, binds each agent on its peer, and forms the execution quorum.
/** Builds the fleet and binds every agent. Port 0 for the console picks a free one. */ public static Seating board(Models models, Options options) throws Exception { int base = options.basePort(); List<Peer> peers = new ArrayList<>(); Peer console = seat(peers, "console", base, 0); Peer triageA = seat(peers, "triage-a", base + 1, base); Peer triageB = seat(peers, "triage-b", base + 2, base); Peer staff = seat(peers, "staff", base + 3, base); Peer riskLead = seat(peers, "risk-lead", base + 4, base); Peer judgeSecurity = seat(peers, "judge-security", base + 5, base); Peer judgeChange = seat(peers, "judge-change", base + 6, base); Peer judgeOps = seat(peers, "judge-ops", base + 7, base); Peer execA = seat(peers, "exec-a", base + 8, base); Peer execB = seat(peers, "exec-b", base + 9, base); Peer execC = seat(peers, "exec-c", base + 10, base); Peer planner = seat(peers, "planner", base + 11, base); CrashSwitch crash = new CrashSwitch(); Triage.Hook hook = options.stageCrash() ? crash : Triage.Hook.NONE; triageA.bind(new Triage("triage-a", models.worker("triage"), hook)); triageB.bind(new Triage("triage-b", models.worker("triage"), hook)); staff.bind(new Assets("assets")); staff.bind(new Remediation("remediation", models.chat("remediation"))); staff.bind(new Operations.Dispatcher()); staff.bind(new Operations.Recorder()); riskLead.bind(new RiskPanel.Lead()); judgeSecurity.bind(new RiskPanel.Judge("judge-security", "security: trust boundaries and credentials", models.chat("judge-security"))); judgeChange.bind(new RiskPanel.Judge("judge-change", "change control: reversibility and maintenance windows", models.chat("judge-change"))); judgeOps.bind(new RiskPanel.Judge("judge-ops", "operations: customer impact and time to recover", models.chat("judge-ops"))); for (OpsFleet.Seat seat : OpsFleet.formExecutionQuorum(List.of(execA, execB, execC))) { seat.peer().bind(new Operations.Executor(seat.raft(), seat.peer().name())); } console.bind(new Operations.WatchOfficer("watch-officer", options.simulatedOperator(), options.reviewTime())); Watchtower watchtower = console.run(Watchtower.over(console)); int consolePort = watchtower.start(options.consolePort()); return new Seating(List.copyOf(peers), console, planner, crash, watchtower, consolePort); }
How the fleet forms and finds itself
All twelve peers join one group named opscenter, whose founding is the text opscenter-v1. The group id comes from that text, the membership policy is OPEN, and the default strategy is LEASE_RACE. The first peer, console, listens on port 7800 with no seed; every other peer dials it on 127.0.0.1:7800 over plain TCP. Ports run from 7800 to 7811 in the order the peers start. Every frame carries a signature (the node's default ChannelAuth.SIGNED) and there is no TLS, which suits a demo on loopback. The library describes an OPEN group as Sybil-exposed by construction, so a fleet that must refuse strangers uses an invitation or a policy validator instead.
console, has no seed and the other eleven dial it.| Setting | Value | Where |
|---|---|---|
| group name, founding | opscenter, the text opscenter-v1 | OpsFleet.GROUP, FOUNDING |
| membership policy, default strategy | OPEN, LEASE_RACE | OpsFleet.group() |
| seed | console at 127.0.0.1:7800; every other peer dials it | OpsCenter.board |
| ports | 7800 to 7811, one per peer in start order; the console server on 7591 | Options.basePort, consolePort |
| transport, channel | plain TcpTransport, every frame signed (ChannelAuth.SIGNED) | OpsFleet.startPeer |
| membership | SWIM, member TTL 30 s, ping timeout 2 s, 2 indirect probes | GroupMembership.Config |
| tick, settle window | 250 ms, 200 ms | OpsFleet.TICK, settleWindow |
Membership runs on SWIM: each peer probes one member per second, waits two seconds for an answer before asking two others to probe, and drops a member that has been silent for 30 seconds. One clock drives all of it. Each peer starts ticking at 250 ms, and that tick advances membership, space replication, votes and the Raft log.
AgentSpaces facade because a vote needs a voter per peer and the ordered log needs one member list, both of which this method and formExecutionQuorum set up directly. One clock, the 250 ms tick, drives membership, replication, votes and the Raft log.public static GroupAdvertisement group() { GroupId groupId = GroupId.fromFounding(FOUNDING.getBytes(StandardCharsets.UTF_8)); return new GroupAdvertisement("aspace://" + groupId.value(), PeerIdentity.generate().peerId(), groupId, Instant.EPOCH, Duration.ofDays(365), GROUP, GroupAdvertisement.MembershipPolicy.OPEN, ConflictStrategyType.LEASE_RACE, GroupAdvertisement.GossipParameters.defaults()); }
/** * Starts one peer. The first peer passes {@code seedPort == 0}; every later peer seeds from * the first. Every peer registers every space and provides a vote capability on the * reviews space, because a ballot's voter is the peer's own agent and ballots that name * anyone else are refused as forged. */ public static Peer startPeer(String name, int port, int seedPort) throws Exception { PeerIdentity identity = PeerIdentity.generate(); PeerNode node = PeerNode.builder(identity).build(); node.listen(new TcpTransport(), HOST + ":" + port); List<PeerAdvertisement.Endpoint> seeds = seedPort == 0 ? List.of() : List.of(new PeerAdvertisement.Endpoint("tcp", HOST + ":" + seedPort, 0)); GroupRuntime runtime = node.joinGroup(group(), new GroupMembership.Config(Duration.ofSeconds(30), Duration.ofSeconds(2), 2), seeds); CborCodec codec = CborCodec.defaultCodec(); InstantSource clock = InstantSource.system(); DiscoveryService discovery = new DiscoveryService(runtime, new AdCache(codec, clock), codec, identity.peerId()); Map<String, ReplicatedSpace> spaces = new LinkedHashMap<>(); for (String spaceName : SPACES) { spaces.put(spaceName, ReplicatedSpace.builder(runtime, spaceName, identity, name) .settleWindow(Duration.ofMillis(200)) .build()); } AgentSpaces facade = new AgentSpaces(identity, clock); AgentSpaces.GroupContext group = facade.register(GROUP, runtime.id(), runtime, discovery) .capabilities(new CapabilityRuntime(runtime, discovery, identity)); spaces.forEach(group::space); VoteCapability vote = new VoteCapability(spaces.get(REVIEWS), identity.agent(name), identity.peerId(), clock); group.provide(vote); // advertised, and wired to @Ballot and @OnDecision on "reviews" node.startTicking(TICK); return new Peer(name, identity, node, runtime, discovery, spaces, facade, group, vote, codec, port); }
How the agents find each other
Every bound agent publishes a signed AgentCard that names what it consumes and produces and which space each type lives in. Fourteen cards circulate here. The agent classes never read a card: the triage worker does not know the proposal writer exists, and each only watches the space and the record type it understands. The cards are for readers that plan, which in this app is the planner peer. It turns the cards into typed actions and routes each call by entry type, so the space delivers the work whether or not a card has arrived.
Why the fleet is wired in code
The Spring Boot starter binds @SpaceTake, @SpaceNotify, @Ballot, @OnDecision and @CapabilityRef beans and provides the vote and aggregate capabilities. Two things in this scenario it does not do. The ordered log runs over a fixed member set that the starter cannot infer, and a quorum needs one voter per peer, so the three judges have to be three peers. The fleet therefore boards through the AgentSpaces facade, the way the numbered examples do, with group.provide(vote) on every peer and group.ordered(space, coordinator) on the three executors. The Party Bus demo makes the same choice.
PEER granularity a peer casts one counted ballot however many agents it hosts, so three judges need three peers. The Raft log runs over a fixed set of members that every member must be given identically, so the three executors name each other when they start. The facade builds both in a few lines, and the Party Bus demo makes the same choice.The vote rule comes from how the capability counts. The tally reads each ballot with its issuer and counts one ballot per issuing peer under the default PEER granularity, which the demo prints as counted per PEER. A ballot also has to name its own writer as voter, or the capability discards it as forged, so each peer builds its own VoteCapability over identity.agent(name). The ordered log gets its member list in formExecutionQuorum, which hands the same three peer ids to every seat before any log starts.
/** * Forms the execution quorum. Every member is given the same fixed member list, and each * registers the coordinator for the actions space, which is what lets {@code @OrderedTake} * bind on that peer. The caller binds the executor agent on each seat. * * <p>Each log keeps its term, vote and entries in memory ({@code MemoryLogStore}, the default), so a * restarted executor rejoins with an empty log. {@code OrderedTakes.over(..., store)} takes a * {@code FileLogStore} per seat when the log must survive a restart; this demo does not, because * its executors live exactly as long as the JVM. */ public static List<Seat> formExecutionQuorum(List<Peer> executors) { List<PeerId> members = executors.stream().map(Peer::peerId).toList(); List<Seat> seats = new ArrayList<>(); long seed = 9000L; for (Peer executor : executors) { OrderedTakes ordered = OrderedTakes.over(executor.group().pipes(), executor.codec(), executor.identity(), executor.name(), members, InstantSource.system(), seed++, executor.space(ACTIONS)); executor.group().ordered(ACTIONS, ordered); // also advertises the Raft log seats.add(new Seat(executor, ordered.raft(), ordered)); } return seats; }
/** An executor seat: its peer, its Raft log and the coordinator that routes its takes through it. */ public record Seat(Peer peer, RaftLog raft, OrderedTakes ordered) { }
Only OpsFleet and OpsCenter.board know about peers, ports, seeds and member lists. The nine agent classes carry annotations and no wiring code.
The six spaces
Every peer holds a replica of all six spaces. Four carry the work: incidents holds what a monitor, a person or a planner raises, evidence holds everything the agents learn, decide and do, reviews holds the vote, and actions holds the orders. The last two, fleet-control and c2-audit, belong to the console: the first for its directives and the second for its audit trail. The console needs fleet-control: whenever command-and-control is on, it shows a "Broadcast directive" form (PAUSE, RESUME, DRAIN) that writes a Directive there, and without the space the form answers "no control space". No agent in this app attaches a DirectiveGate, so a directive is written and audited and nothing honors it; a fleet that wants pause and drain from the console attaches a gate to each worker. Every space declares LEASE_RACE, the default. The executors take from the actions space through an OrderedTakes coordinator, which is how the library offers the ORDERED strategy, since ORDERED is not a setting on a space.
LEASE_RACE, the library's default; the executors take from the actions space through an OrderedTakes coordinator instead, which is how the library offers the ORDERED strategy (it is not a space setting). Nothing here auctions. The first four spaces carry the work; the last two belong to the console, which writes a directive to the first of them and an audit record for every command to the second.| Space and strategy | Entry types | Written by | Read or taken by | Lease |
|---|---|---|---|---|
incidentsLEASE_RACE | Incident | console (12 h), planner (1 h) | triage-a and triage-b take; Assets reacts | take 15 s, entry 12 h |
evidenceLEASE_RACE | Telemetry, Diagnosis, RemediationProposal, RiskReview, RemediationDecision, OperatorApproval, RemediationReceipt, IncidentRecord | every agent, and the console's approval command | reactions subscribe; the console and the recorder read | 12 h |
reviewsLEASE_RACE plus the vote capability | vote Proposal, Ballot | risk-lead opens; the judges cast | the tally on every peer | 12 h |
actionsORDERED through OrderedTakes | RemediationOrder | dispatcher | exec-a, exec-b, exec-c take | take 30 s, entry 12 h |
fleet-controlLEASE_RACE | Directive | the console's Broadcast directive form | nothing: no agent attaches a DirectiveGate | set by the console |
c2-auditLEASE_RACE | C2Record | the console | anyone who audits | set by the console |
Every record carries the incidentId it belongs to. That shared field ties a result to its request for the agents (their templates filter on it), for the recorder (the evidence chain is every entry issued about this incident), and for the Embabel bridge, which pairs a result with its request by matching shared record components. The records are plain Java records.
/** What lands on the board. Written by a monitor, a person at the console, or a planner. */ public record Incident(String incidentId, String source, String summary, String severity) { }
/** The triage agent's conclusion, produced with a model from the summary and the telemetry. */ public record Diagnosis(String incidentId, String cause, double confidence, String diagnosedBy) { }
/** What the remediation planner proposes. Whether it can be undone drives how much coordination it gets. */ public record RemediationProposal(String incidentId, String proposalId, String action, String rationale, boolean reversible, String proposedBy) { }
/** One judge's reasoning, kept on the record beside its signed ballot. */ public record RiskReview(String incidentId, String proposalId, String judge, String option, String reason) { }
/** The outcome of the risk panel's quorum vote, with the tally anyone can recount. */ public record RemediationDecision(String incidentId, String proposalId, String outcome, String tally, String countedPer) { }
/** A person's approval. A signed note like any other, and the gate before the irreversible step. */ public record OperatorApproval(String incidentId, String proposalId, String operator, String note) { }
/** The order on the ordered space. Exactly one executor completes it. */ public record RemediationOrder(String incidentId, String proposalId, String orderId, String action) { }
/** Proof of execution: who ran it and where the Raft log committed the take. */ public record RemediationReceipt(String incidentId, String orderId, String action, String executedBy, long commitIndex) { }
/** The closed incident: outcome plus every piece of evidence, each line naming its signer. */ public record IncidentRecord(String incidentId, String outcome, List<String> evidence) { }
Leases are 12 hours on every entry except the take leases (15 s for triage, 30 s for the executors). A closed incident therefore stays readable in every replica for half a day, which is what lets the console and a late reader see the whole chain.
One run, end to end
OpsCenter.main picks the models from the environment, boards the fleet, waits 2.5 seconds for the cards to gossip, and runs the two incidents. For each one it raises the incident from the console peer and then waits on the evidence space for each stage in turn: diagnosis, proposal, decision, approval, receipt and the closed record, printing the elapsed time as each arrives. The demo reads only; every stage is written by the agent that owns it. If the decision is a rejection, runIncident skips to the closed record, because no approval or receipt will come.
| # | Who | Cue | Writes | Code |
|---|---|---|---|---|
| 1 | console, planner or a monitor | a person or a program | Incident | OpsCenter.raise |
| 2 | staff (Assets); triage-a or triage-b | Incident | Telemetry; Diagnosis | Assets.sample; Triage.diagnose |
| 3 | staff (Remediation) | Diagnosis | RemediationProposal | Remediation.propose |
| 4 | risk-lead | RemediationProposal | vote Proposal remediation:INC-n | Lead.open |
| 5 | judge-security, judge-change, judge-ops | the vote Proposal | three signed ballots; three RiskReviews | Judge.judge |
| 6 | risk-lead | the closed vote | RemediationDecision | Lead.record |
| 7 | console (WatchOfficer) or a person | a RemediationDecision of approve | OperatorApproval | WatchOfficer.review, Watchtower |
| 7a | staff (Recorder) | a RemediationDecision of reject | IncidentRecord, outcome rejected; steps 8 to 10 do not happen | Recorder.closeRejected |
| 8 | staff (Dispatcher) | OperatorApproval, once the panel's approve is on record | RemediationOrder, in actions | Dispatcher.dispatch |
| 9 | exec-a, exec-b or exec-c | RemediationOrder | RemediationReceipt | Executor.execute |
| 10 | staff (Recorder) | RemediationReceipt | IncidentRecord, outcome remediated | Recorder.close |
private static void runIncident(Seating ops, String id, String source, String summary, String severity, boolean killFirstWorker) throws Exception { Space evidence = ops.console().space(OpsFleet.EVIDENCE); long start = System.nanoTime(); raise(ops.console(), id, source, summary, severity); say(id + " written: " + summary); if (killFirstWorker) { if (ops.crash().awaitTake(Duration.ofSeconds(15))) { String victim = ops.crash().victim(); say(victim + " took " + id + " and is consulting the model"); ops.crash().kill(); ops.peer(victim).close(); say("kill -9 " + victim + ". Nothing renews its lease now; it lapses in 15 seconds."); } else { say("no worker took " + id + " within 15 seconds; both stay alive this run"); } } Diagnosis diagnosis = await(() -> first(evidence, Diagnosis.class, id), Duration.ofSeconds(120)) .orElseThrow(() -> new IllegalStateException("no diagnosis for " + id)); say(elapsed(start) + " diagnosis by " + diagnosis.diagnosedBy() + ": " + diagnosis.cause()); RemediationProposal proposal = await(() -> first(evidence, RemediationProposal.class, id), Duration.ofSeconds(60)) .orElseThrow(); say(elapsed(start) + " proposal: " + proposal.action() + " (reversible: " + proposal.reversible() + ")"); RemediationDecision decision = await(() -> first(evidence, RemediationDecision.class, id), Duration.ofSeconds(60)) .orElseThrow(); say(elapsed(start) + " risk panel: " + decision.outcome() + " · tally " + decision.tally() + " · counted per " + decision.countedPer()); if (!RiskPanel.APPROVE.equals(decision.outcome())) { // The panel binds the person: no approval is accepted and no order is placed. The recorder closes it. IncidentRecord rejected = await(() -> first(evidence, IncidentRecord.class, id), Duration.ofSeconds(60)) .orElseThrow(); say(elapsed(start) + " closed as " + rejected.outcome() + ". Evidence chain:"); rejected.evidence().forEach(line -> say(" " + line)); return; } OperatorApproval approval = await(() -> first(evidence, OperatorApproval.class, id), Duration.ofSeconds(600)) .orElseThrow(); say(elapsed(start) + " approved by " + approval.operator() + ": " + approval.note()); RemediationReceipt receipt = await(() -> first(evidence, RemediationReceipt.class, id), Duration.ofSeconds(90)) .orElseThrow(); say(elapsed(start) + " executed once by " + receipt.executedBy() + " at log index " + receipt.commitIndex()); IncidentRecord record = await(() -> first(evidence, IncidentRecord.class, id), Duration.ofSeconds(60)) .orElseThrow(); say(elapsed(start) + " closed. Evidence chain:"); record.evidence().forEach(line -> say(" " + line)); }
public static <T> Optional<T> await(Supplier<Optional<T>> probe, Duration timeout) throws InterruptedException { long deadline = System.nanoTime() + timeout.toNanos(); while (System.nanoTime() < deadline) { Optional<T> found = probe.get(); if (found.isPresent()) { return found; } Thread.sleep(250); } return probe.get(); }
public static <T> List<T> readAll(Space space, Class<T> type, String incidentId) { return space.readAll(Template.of(type).where("incidentId", eq(incidentId)), 20); }
no OPENAI_API_KEY in the environment: the agents answer from the scripted models models: scripted · operator: simulated · console: http://localhost:7591/ (token opscenter-operator) == Incident 1: the disk fills on db-primary, and the first triage worker dies on the job == INC-1 written: disk usage alarm: data volume at 98 percent triage-b took INC-1 and is consulting the model kill -9 triage-b. Nothing renews its lease now; it lapses in 15 seconds. Nov 04, 2026 3:00:02 AM ai.badmonkey.agentspaces.agent.AgentBinder lambda$startTakeLoop$0 WARNING: triage.diagnose: action failed; lease will lapse java.lang.IllegalStateException: triage-b died mid-diagnosis ... [ 16.6s] diagnosis by triage-a: The data volume on db-primary is at 98 percent and the write-ahead log can no longer flush, so writes are stalling [ 17.9s] proposal: Fail over to the standby and rebuild the primary's data volume (reversible: false) [ 19.2s] risk panel: approve · tally approve=2, reject=1 · counted per PEER [ 21.3s] approved by watch-officer: Simulated officer: approved after reading the tally (approve=2, reject=1) [ 21.5s] executed once by exec-a at log index 3 [ 21.5s] closed. Evidence chain: Telemetry · db-primary disk.used.percent=98.4 % · by staff (PEER_ASSERTED) Diagnosis · The data volume on db-primary is at 98 percent and the write-ahead log can no longer flush, so writes are stalling (confidence 0.86) · by triage-a (PEER_ASSERTED) Proposal · Fail over to the standby and rebuild the primary's data volume · reversible: false · by staff (PEER_ASSERTED) Review · judge-ops voted approve: Customer impact outweighs the rebuild cost · by judge-ops (PEER_ASSERTED) Review · judge-change voted reject: An irreversible change outside a maintenance window needs a named owner first · by judge-change (PEER_ASSERTED) Review · judge-security voted approve: No change to the trust boundary · by judge-security (PEER_ASSERTED) Decision · approve · tally approve=2, reject=1 · counted per PEER · by risk-lead (PEER_ASSERTED) Approval · watch-officer: Simulated officer: approved after reading the tally (approve=2, reject=1) · by console (PEER_ASSERTED) Receipt · Fail over to the standby and rebuild the primary's data volume · executed by exec-a · log index 3 · by exec-a (PEER_ASSERTED) == Incident 2: checkout latency, the routine path == INC-2 written: p99 latency over 2 seconds since the 14:10 deploy [ 1.5s] diagnosis by triage-a: The checkout API's connection pool is exhausted after the last deploy doubled its fan-out to the pricing service [ 2.8s] proposal: Roll back checkout-api to the previous release (reversible: true) [ 4.1s] risk panel: approve · tally approve=3, reject=0 · counted per PEER [ 5.9s] approved by watch-officer: Simulated officer: approved after reading the tally (approve=3, reject=0) [ 6.4s] executed once by exec-a at log index 6 [ 6.4s] closed. Evidence chain: Telemetry · checkout-api p99.latency.ms=2350.0 ms · by staff (PEER_ASSERTED) Diagnosis · The checkout API's connection pool is exhausted after the last deploy doubled its fan-out to the pricing service (confidence 0.78) · by triage-a (PEER_ASSERTED) Proposal · Roll back checkout-api to the previous release · reversible: true · by staff (PEER_ASSERTED) Review · judge-change voted approve: Reversible, and the rollback path is known · by judge-change (PEER_ASSERTED) Review · judge-ops voted approve: Low blast radius · by judge-ops (PEER_ASSERTED) Review · judge-security voted approve: No change to the trust boundary · by judge-security (PEER_ASSERTED) Decision · approve · tally approve=3, reject=0 · counted per PEER · by risk-lead (PEER_ASSERTED) Approval · watch-officer: Simulated officer: approved after reading the tally (approve=3, reject=0) · by console (PEER_ASSERTED) Receipt · Roll back checkout-api to the previous release · executed by exec-a · log index 6 · by exec-a (PEER_ASSERTED) INC-1: 1 receipt(s), exactly one expected INC-2: 1 receipt(s), exactly one expected
The seconds vary a little from run to run. What does not vary is the shape: a 15 s gap in INC-1 where the dead worker's lease runs out, a dissent in the INC-1 tally, and one receipt per incident. The last two lines of the output count the receipts.
Leased takes and the model call
Triage is the one agent that takes under the space's own lease race. The parameter type of diagnose becomes a take template on the incidents space, the binder performs the leased take, invokes the method, and when it returns, writes the Diagnosis to the evidence space and completes the take. An exception or a crash only lets the lease lapse, so the incident reappears. The method asks a Spring AI ChatClient for structured output with .entity(Reading.class); Spring AI appends the record's JSON schema to the prompt and binds the reply.
@SpaceTake(space = OpsFleet.INCIDENTS, lease = "15s", pollTimeout = "300ms", resultSpace = OpsFleet.EVIDENCE, resultLease = "12h", description = "Diagnoses an incident: reads the telemetry on record, asks the model for the most " + "likely cause and a confidence, and writes the Diagnosis to the evidence space") public Diagnosis diagnose(Incident incident) { hook.beforeThinking(name, incident); List<Telemetry> readings = evidence.readAll( Template.of(Telemetry.class).where("incidentId", eq(incident.incidentId())), 20); String telemetry = readings.isEmpty() ? "none yet" : readings.stream() .map(t -> t.asset() + " " + t.metric() + "=" + t.value() + " " + t.unit()) .collect(Collectors.joining("; ")); Reading reading = chat.prompt() .system(SYSTEM) .user("Incident " + incident.incidentId() + " from " + incident.source() + " (severity " + incident.severity() + "): " + incident.summary() + "\nTelemetry: " + telemetry) .call() .entity(Reading.class); if (reading == null) { throw new IllegalStateException("the model returned no reading for " + incident.incidentId()); } return new Diagnosis(incident.incidentId(), reading.cause(), reading.confidence(), name); }
/** What the model is asked for. Spring AI appends this record's schema to the prompt. */ public record Reading(String cause, double confidence) { }
ChatClient built over either an OpenAI model or the scripted model, and both answer through the same structured-output path. The triage client carries TakeLeaseAdvisor, which renews the surrounding take on every model round trip, so a slow model does not let a healthy worker's lease lapse and a dead worker still loses the incident one lease after it stops.Models builds one client per role. With OPENAI_API_KEY set, each role gets an OpenAiChatModel (model gpt-4.1-mini unless OPSCENTER_OPENAI_MODEL says otherwise, temperature 0.1). Without it, each role gets a ScriptedChatModel that answers in the JSON shape its agent expects after a think time of 1200 ms (OPSCENTER_THINK_MILLIS overrides it). The triage client alone carries the TakeLeaseAdvisor from agentspaces-springai, which renews the surrounding take on every model round trip. A slow model therefore keeps a healthy worker's claim, and a worker that hangs or dies stops renewing and loses it one lease later.
public ChatClient chat(String role) { return ChatClient.builder(models.apply(role)).build(); } public ChatClient worker(String role) { return ChatClient.builder(models.apply(role)) .defaultAdvisors(new TakeLeaseAdvisor(scheduler, Duration.ofSeconds(30))) .build(); }
static Models fromEnvironment(Map<String, String> env) { String key = env.get(OPENAI_API_KEY); if (key == null || key.isBlank()) { long think = Long.parseLong(env.getOrDefault(THINK_MILLIS, "1200")); return scripted(Duration.ofMillis(think)); } String model = env.getOrDefault(OPENAI_MODEL, "gpt-4.1-mini"); return new Models("openai:" + model, role -> OpenAiChatModel.builder() .options(OpenAiChatOptions.builder().apiKey(key).model(model).temperature(0.1).build()) .build()); }
public ChatResponse call(Prompt prompt) { pause(); String text = prompt.getContents().toLowerCase(Locale.ROOT); return new ChatResponse(List.of(new Generation(new AssistantMessage(answer(text))))); }
The other agents react. Assets and Remediation use @SpaceNotify, which subscribes to a record type without consuming it and writes the return value as the result. Remediation is a reaction rather than a take because a proposal is cheap to recompute, and every planner should see every diagnosis. The model decides whether the action is reversible, and that flag is what the risk panel argues about.
@SpaceNotify(space = OpsFleet.INCIDENTS, lease = "12h", resultSpace = OpsFleet.EVIDENCE, resultLease = "12h", description = "Reports the current reading of the incident's asset as Telemetry in the evidence space") public Telemetry sample(Incident incident) { Reading reading = READINGS.getOrDefault(incident.source(), new Reading("heartbeat", 1, "count")); return new Telemetry(incident.incidentId(), incident.source(), reading.metric(), reading.value(), reading.unit(), name); }
@SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultLease = "12h", description = "Proposes one remediation for a diagnosed incident, with a rationale and whether " + "it can be undone") public RemediationProposal propose(Diagnosis diagnosis) { List<Telemetry> readings = evidence.readAll( Template.of(Telemetry.class).where("incidentId", eq(diagnosis.incidentId())), 20); String telemetry = readings.stream() .map(t -> t.asset() + " " + t.metric() + "=" + t.value() + " " + t.unit()) .collect(Collectors.joining("; ")); Plan plan = chat.prompt() .system(SYSTEM) .user("Diagnosis for " + diagnosis.incidentId() + " (confidence " + diagnosis.confidence() + "): " + diagnosis.cause() + "\nTelemetry: " + telemetry) .call() .entity(Plan.class); if (plan == null) { return null; // nothing to propose; a null reaction writes nothing } return new RemediationProposal(diagnosis.incidentId(), proposalId(diagnosis.incidentId()), plan.action(), plan.rationale(), plan.reversible(), name); }
Delivery is at least once outside the actions space. If a worker is slow rather than dead and a second worker also finishes, two diagnoses can appear. A diagnosis is idempotent, so the cost is a model call.
When a worker dies
The demo stages the failure with CrashSwitch, a hook both triage workers share. Whichever worker takes INC-1 first becomes the victim: it stalls in the hook before it consults the model, and the demo then calls kill() and closes that worker's peer. Closing a peer closes its agents, its facade, its replicas and its node, and the binder interrupts the worker thread, which is how the hook learns it has died. The hook throws, the take never completes, and nobody renews the lease.
/** * Arranges the crash. Both triage workers share this hook. Whichever worker takes the * first incident becomes the victim: it stalls here, before the model is consulted, until * the demo closes its peer. Then the hook throws, so the take never completes, and the * lease lapses with nobody renewing it. Every other call, including the survivor's retry * of the same incident, passes straight through. If nobody kills the victim, the hook * gives up after a minute and the worker carries on. */ public static final class CrashSwitch implements Triage.Hook { private final CountDownLatch armed = new CountDownLatch(1); private volatile boolean killed; private volatile String victim; @Override public void beforeThinking(String worker, Incident incident) { synchronized (this) { if (victim != null) { return; } victim = worker; } armed.countDown(); long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos(); while (!killed && System.nanoTime() < deadline) { try { Thread.sleep(50); } catch (InterruptedException e) { // The binder interrupts its worker when the peer closes: that is the death. Thread.currentThread().interrupt(); killed = true; } } if (killed) { throw new IllegalStateException(worker + " died mid-diagnosis"); } } /** Waits until a worker is holding the first incident. */ public boolean awaitTake(Duration timeout) throws InterruptedException { return armed.await(timeout.toMillis(), TimeUnit.MILLISECONDS); } /** The worker holding the first incident, once {@link #awaitTake} has returned true. */ public String victim() { return victim; } public void kill() { killed = true; } }
if (killFirstWorker) { if (ops.crash().awaitTake(Duration.ofSeconds(15))) { String victim = ops.crash().victim(); say(victim + " took " + id + " and is consulting the model"); ops.crash().kill(); ops.peer(victim).close(); say("kill -9 " + victim + ". Nothing renews its lease now; it lapses in 15 seconds."); } else { say("no worker took " + id + " within 15 seconds; both stay alive this run"); } }
Fifteen seconds after the take, the lease lapses on every replica and the entry is takeable again. The other triage peer polls with a 300 ms timeout, takes it, passes straight through the hook, and finishes in one model call. The recorded run shows it: diagnosis by triage-a at 16.6 s, which is the lease plus the 1.2 s of model time and the time before the first take. The binder logs the dead worker's exception as a warning, triage.diagnose: action failed; lease will lapse, and that line is expected output.
The lease is the recovery time, so a shorter lease recovers faster and needs a worker that renews more often. Here the advisor does the renewing, once per model round trip. The test asserts the outcome that matters: after the kill, the diagnosis comes from the survivor, and the incident still closes.
Three judges and a quorum
Every remediation proposal goes to a risk panel. The lead opens a vote with one annotation, @Propose: the shape of the vote is fixed (options approve and reject, quorum 3, a 12 hour lease) and the cue decides only the question. The binder opens it once per incident, as the lead, and a redelivered proposal opens nothing. Each judge sits on its own peer, asks its own model from its own concern, writes a RiskReview with the reason, and returns an option, which the binder casts as a signed ballot. When three counted ballots are in, the lead's @OnDecision writes the RemediationDecision with the tally.
@Propose. Each judge reads the question, asks its own model, writes the reason and casts a signed ballot from its own peer; the arrows into the tally are those ballots. After three counted ballots the lead's @OnDecision records the outcome with a tally anyone can recount from the signed ballots, so the dissent in INC-1 stays on the record. INC-4 comes from the console session: two judges reject, so the decision is reject./** Opens the vote for each proposal and writes the decision when it closes. */ @AgentSpec(name = "risk-lead", description = "Puts each remediation proposal to the risk panel and records the decision", goals = {"decide remediations"}) public static final class Lead { /** * One vote per remediation, as one annotation (ISSUE-Propose): the shape * is fixed and the cue decides only the question. The binder opens once * per incident, as this agent, and a redelivered proposal opens nothing. */ @Propose(space = OpsFleet.EVIDENCE, vote = OpsFleet.REVIEWS, prefix = PREFIX, key = "incidentId", options = {"approve", "reject"}, quorum = QUORUM, lease = "12h", subscriptionLease = "12h", description = "Opens a risk-panel vote on a remediation proposal, one vote per incident") public String open(RemediationProposal proposal) { return "Approve remediation for " + proposal.incidentId() + ": " + proposal.action() + ". Reversible: " + (proposal.reversible() ? "yes" : "no") + ". Rationale: " + proposal.rationale(); } @OnDecision(space = OpsFleet.REVIEWS, prefix = PREFIX, lease = "12h", resultSpace = OpsFleet.EVIDENCE, resultLease = "12h") public RemediationDecision record(VoteCapability.Decision decision) { String incidentId = incidentOf(decision.proposalId()); String tally = new TreeMap<>(decision.tally()).entrySet().stream() .map(e -> e.getKey() + "=" + e.getValue()) .collect(Collectors.joining(", ")); return new RemediationDecision(incidentId, Remediation.proposalId(incidentId), decision.winner(), tally, decision.granularity().name()); } }
@Ballot(space = OpsFleet.REVIEWS, prefix = PREFIX, lease = "12h") public String judge(VoteCapability.Proposal proposal) { Verdict verdict = chat.prompt() .system(SYSTEM) .user("Your concern: " + concern + "\nProposal: " + proposal.question() + "\nOptions: " + String.join(", ", proposal.options())) .call() .entity(Verdict.class); if (verdict == null || !proposal.options().contains(verdict.option())) { return null; // abstain rather than cast an option the proposal does not offer } String incidentId = incidentOf(proposal.proposalId()); evidence.write(new RiskReview(incidentId, Remediation.proposalId(incidentId), seat, verdict.option(), verdict.reason()), Lease.of(Duration.ofHours(12))); return verdict.option(); }
private String judge(String text) { // Judge the proposal, not the judge's own brief: the concern line names its topics too. int at = text.indexOf("proposal:"); String proposal = at >= 0 ? text.substring(at) : text; boolean irreversible = proposal.contains("reversible: no"); return switch (role) { case "judge-change" -> irreversible ? json("option", "reject", "reason", "An irreversible change outside a maintenance window needs a named owner first") : json("option", "approve", "reason", "Reversible, and the rollback path is known"); case "judge-security" -> proposal.contains("credential") ? json("option", "reject", "reason", "Touches credentials; needs a security review") : json("option", "approve", "reason", "No change to the trust boundary"); default -> json("option", "approve", "reason", irreversible ? "Customer impact outweighs the rebuild cost" : "Low blast radius"); }; }
The scripted judges make INC-1 a genuine disagreement. The change-control judge rejects any proposal marked reversible: no, and the fail-over is irreversible, so the tally is two to one. The panel still approves. The dissent stays on the record twice: as a RiskReview with the reason, and as a signed ballot in the reviews space that anyone can recount. A third scripted incident goes the other way. For the key exposed in a public repository, the proposed purge is irreversible, so the change-control judge rejects it, and it touches a credential, so the security judge rejects it too. The operations judge approves, the tally is approve=1, reject=2, and the decision is reject. The next chapter shows what a rejection does.
A quorum proves that three judges agreed, and nothing more. The scripted judges are rules; live models can all be wrong together. The vote also counts peers, which is why the three judges are three peers, and a judge that returns null abstains rather than cast an option the proposal does not offer.
The human gate
A remediation does not run on the panel's say alone, and it does not run on a person's say alone either. The dispatcher reacts to an OperatorApproval and to nothing else, and the gate lets a person approve only what the panel approved. Two paths write the approval. Headless, the WatchOfficer on the console peer approves a decision of approve after a 2 second review and says so in the note. With OPSCENTER_OPERATOR=console, the officer returns null and the run waits for a person to press "Approve remediation" in the console.
OperatorApproval, and the dispatcher asks RiskPanel.refusal once more before it places the order. After a reject decision the officer answers nothing, the console refuses the command and audits the refusal, and the recorder closes the incident as rejected. An approval entry written around the console reaches the dispatcher, which refuses it for the same reason.The panel binds the person in three places. The console command asks RiskPanel.refusal before it writes anything. For a rejected incident it answers HTTP 400 with the reason and the tally, for an incident the panel has not decided it says so, and in both cases it writes a c2-audit record of the refusal. The dispatcher asks the same question when an approval reaches it, because any peer can write an OperatorApproval entry and the console is only one of them. A decision that has not reached the staff peer's replica yet is waited for up to five seconds and then counts as no decision. When the panel rejected, the officer answers nothing and the recorder's second cue closes the incident as rejected.
/** * Why an operator approval of this incident must be refused, or empty when the panel passed it. * The human gate sits behind the panel: a person approves what the quorum approved and nothing * else, so a remediation the panel rejected never reaches the executors on a human's say. The * console command and the dispatcher both ask this, because an approval entry can also arrive * from a peer that is not the console. A decision that has not reached this replica yet is * waited for up to {@code wait}; a decision that is not there by then counts as no decision. */ public static Optional<String> refusal(Space evidence, String incidentId, Duration wait) { long deadline = System.nanoTime() + wait.toNanos(); Optional<RemediationDecision> decision = decisionOn(evidence, incidentId); while (decision.isEmpty() && System.nanoTime() < deadline) { try { Thread.sleep(100); } catch (InterruptedException e) { Thread.currentThread().interrupt(); break; } decision = decisionOn(evidence, incidentId); } if (decision.isEmpty()) { return Optional.of("the risk panel has not decided on " + incidentId + " yet"); } RemediationDecision decided = decision.get(); return APPROVE.equals(decided.outcome()) ? Optional.empty() : Optional.of("the risk panel rejected " + incidentId + " (" + decided.tally() + ")"); }
@SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultLease = "12h", description = "Approves a remediation the risk panel approved, after a review delay; " + "answers nothing to a rejection, and nothing at all when a person staffs the console") public OperatorApproval review(RemediationDecision decision) { if (!simulated || !RiskPanel.APPROVE.equals(decision.outcome())) { return null; // a rejection ends here; a staffed center waits for the console } try { Thread.sleep(reviewTime); } catch (InterruptedException e) { Thread.currentThread().interrupt(); return null; } return new OperatorApproval(decision.incidentId(), decision.proposalId(), operator, "Simulated officer: approved after reading the tally (" + decision.tally() + ")"); }
private static ConsoleCommand approveRemediation(Space evidence) { return new ConsoleCommand() { @Override public String id() { return "approve-remediation"; } @Override public String title() { return "Approve remediation"; } @Override public String description() { return "The human gate. Approves a remediation the risk panel approved; the dispatcher then " + "places the order on the ordered actions space. An incident the panel rejected, " + "or has not decided, is refused."; } @Override public List<Field> fields() { return List.of(Field.text("incidentId", "Incident id"), Field.text("note", "Note")); } @Override public Outcome execute(Map<String, String> args, CommandContext ctx) { String id = args.getOrDefault("incidentId", "").trim(); if (id.isEmpty()) { return Outcome.failed("name the incident to approve"); } Optional<String> refusal = RiskPanel.refusal(evidence, id, Duration.ZERO); if (refusal.isPresent()) { ctx.audit("approve-remediation refused", id + ": " + refusal.get()); return Outcome.failed("approval refused: " + refusal.get()); } OperatorApproval approval = new OperatorApproval(id, Remediation.proposalId(id), ctx.operator(), args.getOrDefault("note", "approved from the console")); String entryId = ctx.dispatch(OpsFleet.EVIDENCE, approval, Duration.ofHours(12)); ctx.audit("approve-remediation", id + " by " + ctx.operator()); return Outcome.dispatched("approved " + id, entryId); } }; }
@SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultSpace = OpsFleet.ACTIONS, resultLease = "12h", description = "Places an operator-approved remediation on the ordered actions space as an order; " + "refuses an approval for a remediation the risk panel did not approve") public RemediationOrder dispatch(OperatorApproval approval) { Optional<String> refusal = RiskPanel.refusal(evidence, approval.incidentId(), DECISION_WAIT); if (refusal.isPresent()) { LOG.warning("dispatcher refuses the approval of " + approval.incidentId() + ": " + refusal.get()); return null; // no order, so no executor ever sees it } Optional<RemediationProposal> proposal = evidence.readAll( Template.of(RemediationProposal.class).where("incidentId", eq(approval.incidentId())), 1) .stream().findFirst(); return proposal.map(p -> new RemediationOrder(approval.incidentId(), approval.proposalId(), "order-" + approval.incidentId(), p.action())).orElse(null); }
@SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultLease = "12h", description = "Closes an incident the risk panel rejected as rejected, with the evidence " + "gathered up to the decision; writes nothing for an approved decision") public IncidentRecord closeRejected(RemediationDecision decision) { if (RiskPanel.APPROVE.equals(decision.outcome())) { return null; // the receipt closes an approved incident } awaitReviews(decision.incidentId()); return record(decision.incidentId(), REJECTED); }
The check lives in the console command and the dispatcher. It does not make the actions space refuse an order: any peer admitted to the group can write a RemediationOrder, as any peer can write any entry in this open demo group, and a deployment that must stop that restricts which peers may write that space. The approval itself is still one signed note that names an operator; the single static token audits as operator, and an OIDC validator would put each person's own subject in that field.
The console: stages and commands
The fleet console is the library's FleetConsoleServer, served from the console peer at http://localhost:7591/. Watchtower gives it the stock views (members, advertisements, the incidents, actions and evidence spaces with per-worker completion counts), an Incidents panel of its own, and two commands behind the operator token opscenter-operator. A command runs through FleetCommander under the console peer's key, so what it writes is a signed, leased entry, and the commander also writes a c2-audit record that names the command and the operator. With a single static token the library audits every holder as operator.
decided: reject and then closed: rejected. A triage worker's completed take removes the Incident from the board, so the console peer keeps a roster of every incident it saw written and reads the stage from the evidence space.The panel answers one question per incident: how far has it got. It reads the stage from the evidence space, from the end of the lifecycle backwards, so the first record that exists names the stage. The incident itself needs more care. A triage worker's completed take removes the Incident from the board, and a row built only from the incidents space would exist until the first claim and then vanish. The panel therefore keeps a Roster: a notify subscription on the console peer's incidents space that remembers each incident written, which also supplies the source and the severity for the row. A rejected remediation reads decided: reject and then closed: rejected.
/** * What the console peer has seen written to the incidents space. A triage worker takes an * incident, and a completed take removes the entry, so the space alone shows an incident * only until the first worker claims it. The roster keeps each one for the panel, which * then reads the stage from the evidence space. */ private static final class Roster implements AutoCloseable { private static final int CAPACITY = 500; private final Space incidents; private final Map<String, Incident> seen = Collections.synchronizedMap(new LinkedHashMap<>() { @Override protected boolean removeEldestEntry(Map.Entry<String, Incident> eldest) { return size() > CAPACITY; } }); private final Subscription subscription; Roster(Space incidents) { this.incidents = incidents; this.subscription = incidents.notify(Template.of(Incident.class), event -> { if (event.kind() == SpaceEvent.Kind.WRITTEN) { seen.putIfAbsent(event.entry().incidentId(), event.entry()); } }, Lease.of(Duration.ofDays(365))); } /** Every incident this console has seen, oldest first, including any still unclaimed on the board. */ List<Incident> all() { for (Incident incident : incidents.readAll(Template.of(Incident.class), 50)) { seen.putIfAbsent(incident.incidentId(), incident); } synchronized (seen) { return List.copyOf(seen.values()); } } @Override public void close() { subscription.close(); } }
/** The furthest stage the evidence space shows for the incident, from the end of the lifecycle backwards. */ private static String stage(Space evidence, String id) { Optional<IncidentRecord> record = find(evidence, IncidentRecord.class, id); if (record.isPresent()) { return "closed: " + record.get().outcome(); } if (find(evidence, RemediationReceipt.class, id).isPresent()) { return "executed"; } if (find(evidence, OperatorApproval.class, id).isPresent()) { return "approved"; } Optional<RemediationDecision> decision = find(evidence, RemediationDecision.class, id); if (decision.isPresent()) { return "decided: " + decision.get().outcome() + " (" + decision.get().tally() + ")"; } if (find(evidence, RemediationProposal.class, id).isPresent()) { return "under review"; } if (find(evidence, Diagnosis.class, id).isPresent()) { return "diagnosed"; } return "open"; }
The two commands are "Raise incident", which writes an Incident the way a monitor would (its source list includes payments-vault), and "Approve remediation", the gate above. The page also carries a "Broadcast directive" form that the library adds whenever command-and-control is on. It writes a Directive into fleet-control, and nothing in this app honors it, because no agent attaches a DirectiveGate.
[ 0.0s] demo started [ 4.3s] panel INC-1 db-primary high open [ 4.5s] POST raise-incident {"incidentId":"INC-4","source":"payments-vault","summary":"service account key exposed in a public repository","severity":"critical"} -> 200 {"ok":true,"message":"raised INC-4","entryId":"4684c6c6-77ab-45bc-9495-169f065c3ddf","operator":"operator"} [ 4.5s] panel INC-4 payments-vault critical open [ 6.1s] panel INC-4 payments-vault critical diagnosed [ 7.3s] panel INC-4 payments-vault critical under review [ 8.6s] panel INC-4 payments-vault critical closed: rejected [ 8.6s] POST approve-remediation {"incidentId":"INC-4","note":"approve it anyway"} -> 400 {"ok":false,"message":"approval refused: the risk panel rejected INC-4 (approve=1, reject=2)","operator":"operator"} [ 8.6s] POST approve-remediation {"incidentId":"INC-99","note":"no such incident"} -> 400 {"ok":false,"message":"approval refused: the risk panel has not decided on INC-99 yet","operator":"operator"} [ 20.8s] panel INC-1 db-primary high diagnosed [ 21.9s] panel INC-1 db-primary high under review [ 23.2s] panel INC-1 db-primary high decided: approve (approve=2, reject=1) [ 23.3s] POST approve-remediation {"incidentId":"INC-1","note":"approved from the console after reading the tally"} -> 200 {"ok":true,"message":"approved INC-1","entryId":"febc96b4-3a01-4e6f-a6fb-49e24232818a","operator":"operator"} [ 23.3s] POST directive PAUSE triage-a -> 200 {"ok":true,"message":"PAUSE sent to triage-a","operator":"operator"} [ 23.4s] panel INC-1 db-primary high approved [ 23.7s] panel INC-1 db-primary high closed: remediated [ 23.8s] panel INC-2 checkout-api medium open [ 25.3s] panel INC-2 checkout-api medium diagnosed [ 26.5s] panel INC-2 checkout-api medium under review [ 27.8s] panel INC-2 checkout-api medium decided: approve (approve=3, reject=0) [ 27.8s] POST approve-remediation {"incidentId":"INC-2","note":"approved from the console after reading the tally"} -> 200 {"ok":true,"message":"approved INC-2","entryId":"f69286c5-71e4-45f6-a0f2-2c4229013b9b","operator":"operator"} [ 27.9s] panel INC-2 checkout-api medium approved [ 28.3s] panel INC-2 checkout-api medium closed: remediated [ 28.3s] all three incidents closed [ 30.6s] demo stopped
OPSCENTER_OPERATOR=console (the script starts the demo as mvn exec:java -Dexec.args=--watch), polls the Incidents panel every 0.1 s and presses the commands over HTTP with the operator token; the lines are the script's own outputRead the session from the top. The script raised INC-4 through the console command at 4.5 s, while INC-1's first worker was dead and its lease had not yet lapsed, so the other triage worker took INC-4 and it reached closed: rejected at 8.6 s. The script then pressed Approve on it, and the console answered HTTP 400 with the reason and the tally. It also pressed Approve on INC-99, which the panel never decided, and got the matching 400. INC-1 reappeared when its lease lapsed, and the script approved it the moment the panel showed decided: approve, at 23.2 s. The PAUSE directive came back 200 and changed nothing, because no agent attaches a gate to fleet-control. The panel never listed executed, because the recorder closes the incident as soon as the receipt lands, so that stage falls between two of the script's 0.1 s polls.
The panel is a view over the console peer's replica, so a row can lag the fleet by a replication round. The stage names a record that exists, and it does not name who is working on the incident: while a worker holds an incident under its take lease the row still says open. The roster holds the last 500 incidents the console saw written; a record whose incident the console never saw still gets a row, with an empty source.
Exactly once: the executors
The dispatcher writes a RemediationOrder to the actions space, which every peer replicates. Three executor peers each run an @OrderedTake loop on it. The take goes through an OrderedTakes coordinator rather than the space's own claim race. Each executor that sees the order submits a signed claim as a Raft command, the log applies the first claim for the entry's epoch and rejects the rest, and the winner adopts the entry, runs execute, and completes with a receipt. The same decision falls out on every member because it depends only on the committed log.
/** * The one step that cannot happen twice. Three executors race for every order and the * Raft log commits exactly one claimant. The receipt carries the commit index, so the * record shows where in the log the action was committed. */ @AgentSpec(name = "executor", description = "Executes an approved remediation exactly once", goals = {"execute remediations"}) public static final class Executor { private final RaftLog raft; private final String name; public Executor(RaftLog raft, String name) { this.raft = raft; this.name = name; } @OrderedTake(space = OpsFleet.ACTIONS, lease = "30s", pollTimeout = "2s", resultSpace = OpsFleet.EVIDENCE, resultLease = "12h", description = "Executes an order exactly once: the Raft log commits one claimant, and the " + "receipt carries the commit index") public RemediationReceipt execute(RemediationOrder order) { try { Thread.sleep(300); // the work itself } catch (InterruptedException e) { Thread.currentThread().interrupt(); } return new RemediationReceipt(order.incidentId(), order.orderId(), order.action(), name, raft.commitIndex()); } }
public static List<Seat> formExecutionQuorum(List<Peer> executors) { List<PeerId> members = executors.stream().map(Peer::peerId).toList(); List<Seat> seats = new ArrayList<>(); long seed = 9000L; for (Peer executor : executors) { OrderedTakes ordered = OrderedTakes.over(executor.group().pipes(), executor.codec(), executor.identity(), executor.name(), members, InstantSource.system(), seed++, executor.space(ACTIONS)); executor.group().ordered(ACTIONS, ordered); // also advertises the Raft log seats.add(new Seat(executor, ordered.raft(), ordered)); } return seats; }
The receipt records raft.commitIndex() at the moment the executor builds it. In the recorded run INC-1 reports log index 3 and INC-2 reports 6. Each executor that sees an order submits a claim command, and the index the receipt carries is the commit index at the moment the winner builds it, so it depends on how many claims had committed by then and differs from run to run. The winner was exec-a for INC-1 and exec-a for INC-2; it can vary, and the count of receipts does not. The demo prints the receipts and the test asserts one per incident, after a two second pause that gives any duplicate time to appear.
Exactly once costs liveness on a minority. A member that cannot reach a majority of the three cannot commit a claim, so it cannot take. Two of three executors must be up for any order to run. The log here lives in memory (the default MemoryLogStore), so a restarted executor rejoins with an empty log. OrderedTakes.over(..., store) takes a FileLogStore per seat, which keeps the term, the vote and the entries on disk, so an executor can rejoin with its vote intact after a restart. The demo keeps the default because its executors live exactly as long as the JVM. A remediation the panel rejected never reaches this chapter: no order is written, so no executor claims anything.
The evidence chain
The recorder reacts to the receipt and closes the incident. It assembles the record from what the space already holds: for each record type, it reads every entry about the incident with readAllIssued, which returns the entry together with its issuer and its attestation, and prints each line with the peer that signed it. It has two cues. A RemediationReceipt closes the incident as remediated. A RemediationDecision of reject closes it as rejected, with the same kinds of line up to the decision and no approval or receipt, after waiting up to three seconds for the three reviews to reach its replica, because the decision is written the moment the third ballot is counted.
/** * Closes the incident. The record is assembled from what the space already holds: every * entry about the incident, read with its issuer, so each line names who signed it and * how strongly. Nothing here is narrated after the fact. Every line reads {@code PEER_ASSERTED} * because each peer signs with its own key and vouches for the agents it hosts; an agent * with a certified key of its own ({@code PeerIdentity.subordinate}) would read * {@code AGENT_ATTESTED} and name itself. An executed remediation closes as * {@code remediated}; one the risk panel rejected closes as {@code rejected}, with the * evidence gathered up to the decision and no approval or receipt in it. */ @AgentSpec(name = "recorder", description = "Closes an incident with its full evidence chain", goals = {"record incidents"}) public static final class Recorder { public static final String REMEDIATED = "remediated"; public static final String REJECTED = "rejected"; /** How long the judges' reviews may lag behind the decision on this peer's replica. */ private static final Duration REVIEW_WAIT = Duration.ofSeconds(3); @SpaceRef(OpsFleet.EVIDENCE) Space evidence; @SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultLease = "12h", description = "Closes an executed incident as remediated, with the evidence chain: " + "every entry about it, each line naming the peer that signed it") public IncidentRecord close(RemediationReceipt receipt) { return record(receipt.incidentId(), REMEDIATED); } @SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultLease = "12h", description = "Closes an incident the risk panel rejected as rejected, with the evidence " + "gathered up to the decision; writes nothing for an approved decision") public IncidentRecord closeRejected(RemediationDecision decision) { if (RiskPanel.APPROVE.equals(decision.outcome())) { return null; // the receipt closes an approved incident } awaitReviews(decision.incidentId()); return record(decision.incidentId(), REJECTED); } /** The decision is written once all ballots are in, and each review precedes its ballot; let the replica catch up. */ private void awaitReviews(String incidentId) { long deadline = System.nanoTime() + REVIEW_WAIT.toNanos(); while (reviews(incidentId) < RiskPanel.QUORUM && System.nanoTime() < deadline) { try { Thread.sleep(100); } catch (InterruptedException e) { Thread.currentThread().interrupt(); return; } } } private int reviews(String incidentId) { return evidence.readAll(Template.of(RiskReview.class).where("incidentId", eq(incidentId)), 50).size(); } private IncidentRecord record(String id, String outcome) { List<String> lines = new ArrayList<>(); lines.addAll(issued(Telemetry.class, id, t -> "Telemetry · " + t.asset() + " " + t.metric() + "=" + t.value() + " " + t.unit())); lines.addAll(issued(Diagnosis.class, id, d -> "Diagnosis · " + d.cause() + " (confidence " + d.confidence() + ")")); lines.addAll(issued(RemediationProposal.class, id, p -> "Proposal · " + p.action() + " · reversible: " + p.reversible())); lines.addAll(issued(RiskReview.class, id, r -> "Review · " + r.judge() + " voted " + r.option() + ": " + r.reason())); lines.addAll(issued(RemediationDecision.class, id, d -> "Decision · " + d.outcome() + " · tally " + d.tally() + " · counted per " + d.countedPer())); lines.addAll(issued(OperatorApproval.class, id, a -> "Approval · " + a.operator() + ": " + a.note())); lines.addAll(issued(RemediationReceipt.class, id, r -> "Receipt · " + r.action() + " · executed by " + r.executedBy() + " · log index " + r.commitIndex())); return new IncidentRecord(id, outcome, List.copyOf(lines)); } private <T> List<String> issued(Class<T> type, String incidentId, Function<T, String> describe) { return evidence.readAllIssued(Template.of(type).where("incidentId", eq(incidentId)), 50).stream() .map(i -> describe.apply(i.entry()) + " · by " + i.issuer().localName() + " (" + i.attestation() + ")") .toList(); } }
Every line in the recorded run ends in PEER_ASSERTED. The peer's key signed the entry and the peer named the agent. The staff peer hosts four agents, so its two lines, the telemetry and the proposal, both read by staff. Giving each agent a certified key of its own, as example 13 does with PeerIdentity.subordinate, would make those lines AGENT_ATTESTED and name Assets and Remediation separately. Per-agent keys would also let a vote count per agent, which the demo's granularity does not.
The record is a snapshot: the recorder reads its own replica when the receipt arrives, and the record is itself an entry with a 12 hour lease. It reports the incident and does not archive it.
The planner seat: Embabel over the fleet
The twelfth peer has no agents. It reads the fleet's cards. RemoteActions turns each foreign card into an invocable action: invoking the triage action writes an Incident into the incidents space and waits for the Diagnosis that shares its incidentId. The planner holds the incidents, evidence and actions spaces, a route that sends incidents to incidents, and a result route for every type the fleet produces. RemoteActions waits for a result in the space of the input unless a result route says otherwise, so the table in Bridge.RESULT_SPACES decides whether an invocation finds its answer: the assets action takes an Incident from incidents and answers with a Telemetry in evidence, and the dispatcher action takes an OperatorApproval from evidence and answers with a RemediationOrder in actions. The first fleet test invokes the triage action with a fresh incident, INC-3, checks that every generated action awaits its result in the space the table names and carries a description, and invokes the assets action to watch its telemetry arrive in evidence. The second fleet test invokes the dispatcher action and receives its order from the actions space.
/** * The space each agent writes its result into, by result type, which is where an invocation * must wait for it: RemoteActions waits in the input's space unless told otherwise. Every * action the fleet's cards generate has its output here. Triage, assets, remediation, the * officer, the executors and the recorder answer in evidence; the dispatcher answers in * actions, so the planner holds that space too. */ public static final Map<Class<?>, String> RESULT_SPACES = Map.of( Diagnosis.class, OpsFleet.EVIDENCE, // Triage Telemetry.class, OpsFleet.EVIDENCE, // Assets RemediationProposal.class, OpsFleet.EVIDENCE, // Remediation OperatorApproval.class, OpsFleet.EVIDENCE, // WatchOfficer RemediationOrder.class, OpsFleet.ACTIONS, // Dispatcher: resultSpace = actions RemediationReceipt.class, OpsFleet.EVIDENCE, // Executor IncidentRecord.class, OpsFleet.EVIDENCE); // Recorder, both of its closes
public static RemoteActions remoteActions(OpsFleet.Peer planner) { Map<String, Space> spaces = Map.of(OpsFleet.INCIDENTS, planner.space(OpsFleet.INCIDENTS), OpsFleet.EVIDENCE, planner.space(OpsFleet.EVIDENCE), OpsFleet.ACTIONS, planner.space(OpsFleet.ACTIONS)); RemoteActions remote = new RemoteActions(planner.discovery(), planner.peerId(), spaces) .writeLease(Lease.of(Duration.ofHours(1))) .route(Incident.class, OpsFleet.INCIDENTS); RESULT_SPACES.forEach(remote::resultsIn); return remote; }
public static EmbabelRemoteActions bridge(RemoteActions remote) { return new EmbabelRemoteActions(remote, INVOKE_TIMEOUT, AGENT_NAME, "The operations center fleet's advertised skills: triage, assets, remediation, " + "dispatch, officer review, execution and recording"); }
/** * Waits for the one action a plan needs. The bridge generates its agent from the actions * available at that moment, so bridge only once the card you plan over has arrived. */ public static Optional<RemoteAction> awaitAction(RemoteActions remote, Class<?> inputType, Class<?> outputType, Duration timeout) throws InterruptedException { long deadline = System.nanoTime() + timeout.toNanos(); while (System.nanoTime() < deadline) { Optional<RemoteAction> found = remote.consuming(inputType).stream() .filter(a -> outputType.equals(a.outputType())).findFirst(); if (found.isPresent()) { return found; } Thread.sleep(200); } return Optional.empty(); }
Bridge.RESULT_SPACES names for its output type, shown in parentheses. That is why the planner holds the actions space: only the dispatcher's order (shaded) appears there. This is the list once the fleet has settled; the demo bridges as soon as the triage card arrives.EmbabelRemoteActions then generates an Embabel @Agent class whose methods are those actions, one typed @Action per entry, named from the card and the consumed type. It reads Embabel's annotations by name, so the app compiles without Embabel, and under -Pembabel it deploys the generated agent onto the platform beside the local one. The local planner knows two actions, raise (report to incident) and brief (diagnosis to briefing), and cannot connect them. The fleet's triage_Incident closes the gap.
@Agent(description = "Turns an incident report into a briefing with the operations center fleet") public static class OpsPlannerAgent { @Action public Incident raise(IncidentReport report) { return new Incident(report.incidentId(), report.source(), report.summary(), report.severity()); } @AchievesGoal(description = "A briefing on the incident's most likely cause") @Action public Briefing brief(Diagnosis diagnosis) { return new Briefing(diagnosis.incidentId(), diagnosis.cause(), diagnosis.confidence(), diagnosis.diagnosedBy()); } }
RemoteActions remote = Bridge.remoteActions(ops.planner()); Bridge.awaitTriage(remote, Duration.ofSeconds(60)) .orElseThrow(() -> new IllegalStateException("the triage card did not arrive")); System.out.println("fleet actions discovered: " + remote.available().size() + ", including triage (Incident -> Diagnosis)"); EmbabelRemoteActions bridge = Bridge.bridge(remote); if (!bridge.deployTo(platform)) { throw new IllegalStateException("the fleet agent did not deploy; are the cards in?"); } // One planning space over the local planner and the generated fleet agent. com.embabel.agent.core.Agent planner = platform.createAgent("opscenter-planner", "opscenter", "Briefs incidents with the operations center fleet's specialists"); AgentProcess process = platform.createAgentProcessFrom(planner, ProcessOptions.DEFAULT, new IncidentReport("INC-7", "auth-gateway", "error rate above 10 percent on token refresh", "high")); process.run();
public static void main(String[] args) { // This process is the planner only. The fleet is boarded in code and bridged by hand, // so neither the AgentSpaces starter nor its Embabel auto-configuration (which binds // @Agent beans through the starter's peer) may stand up a peer of its own here. new SpringApplicationBuilder(OpsCenterEmbabelApplication.class) .properties(Map.of( "spring.main.web-application-type", "none", "spring.autoconfigure.exclude", "ai.badmonkey.agentspaces.spring.AgentSpacesAutoConfiguration," + "ai.badmonkey.agentspaces.spring.AgentSpacesConsoleAutoConfiguration," + "ai.badmonkey.agentspaces.spring.AgentSpacesA2aAutoConfiguration," + "ai.badmonkey.agentspaces.embabel.EmbabelAgentSpacesAutoConfiguration")) // Embabel ships its own default-llm, which outranks builder properties; a // command-line argument outranks both. .run(withDefaultLlm(args)); }
raise -> triage_Incident -> brief, the plan the code printed, and plans again after each step. The middle step runs on the fleet.Building and running the profile at 0.5.0
The Embabel sources sit in src/embabel/java and compile only under -Pembabel, which adds embabel-agent-starter 1.5.2 and Spring Boot 4.1.1. On 2026-11-04 mvn -Pembabel compile built against AgentSpaces 0.5.0 with BUILD SUCCESS; javac warns that @EnableAgents is deprecated and marked for removal. The application ran with scripted models and printed the plan below. It excludes the AgentSpaces auto-configurations on purpose, because it boards the fleet in code rather than through the starter, and it waits for the triage card before it bridges, because the generated agent is a snapshot of the actions available at that moment. In the run recorded here the other cards had already arrived, so that snapshot held all 11 actions and the generated agent has a method for each. The planner also logs an ERROR line for each generated action that Embabel cannot reach as a goal (8 lines in this run). Every generated action carries @AchievesGoal, because its card names goals, so Embabel treats each one as a candidate goal and reports the ones whose input nothing in the planning space produces: watch_officer_RemediationDecision and recorder_RemediationDecision, whose RemediationDecision input only the risk panel writes. The plan to the briefing is unaffected, and the block below leaves those lines out.
console: http://localhost:7591/ fleet actions discovered: 11, including triage (Incident -> Diagnosis) Embabel - Deployed agent opsCenterFleet description: The operations center fleet's advertised skills: triage, assets, remediation, dispatch, officer review, execution and recording Embabel - [elegant_rosalind] formulated plan: OpsPlannerAgent.raise -> RemoteFleetAgent$45a155a9.triage_Incident -> OpsPlannerAgent.brief Embabel - [elegant_rosalind] executed action OpsPlannerAgent.raise in PT0.001S Embabel - [elegant_rosalind] formulated plan: RemoteFleetAgent$45a155a9.triage_Incident -> OpsPlannerAgent.brief Embabel - [elegant_rosalind] executed action RemoteFleetAgent$45a155a9.triage_Incident in PT2.032S Embabel - [elegant_rosalind] formulated plan: OpsPlannerAgent.brief Embabel - [elegant_rosalind] executed action OpsPlannerAgent.brief in PT0S Embabel - [elegant_rosalind] goal OpsPlannerAgent.brief achieved in PT2.068105S briefing: Briefing[incidentId=INC-7, cause=The auth gateway's token-refresh path is failing because its upstream signing credential expired at midnight, confidence=0.74, diagnosedBy=triage-a]
raise -> triage_Incident -> brief; timestamps, package names and the world-state dumps are left outmvn -Pembabel compile exec:java -Dopscenter.main=ai.badmonkey.agentspaces.opscenter.embabel.OpsCenterEmbabelApplication
The incident the planner raises lands in the same space as any other, so the rest of the center sees it. Without --watch the process closes the fleet right after the briefing, so the proposal, vote and execution stages have only begun. Run with --watch to keep the fleet up while the center works the incident, and add OPSCENTER_OPERATOR=console to approve it yourself. Every generated action also carries @AchievesGoal, because the card names goals, which gives the planner more than one goal to choose from once the whole catalogue is bridged.
Running, testing, and where to go next
The project is a standalone Maven build. agentspaces-dependencies 0.5.0 from Maven Central manages the core artifacts and embabel-agentspaces, and agentspaces-springai 0.5.0 is pinned in the pom because the BOM does not manage it. No checkout of the core is needed.
./run-demo.sh # scripted models, simulated operator, exits when both incidents close ./run-demo.sh --watch # the same, then keeps the fleet and the console up OPSCENTER_OPERATOR=console ./run-demo.sh --watch # a person approves from the console OPENAI_API_KEY=sk-... ./run-demo.sh # live models through Spring AI's OpenAI client mvn test
The console is at http://localhost:7591/ with the operator token opscenter-operator. Peers use ports 7800 to 7811 on loopback, so run one demo at a time. run-demo.sh compiles first, because the project sits outside the core reactor and nothing else builds its classes.
What the test asserts
mvn test runs 7 tests. ScriptedModelsTest has 5, and checks that the scripted models answer through Spring AI's structured-output path, including the judges' dissent and the exposed-key scenario. OpsCenterFleetTest has two, and each runs a whole fleet over loopback TCP on a free block of twelve ports, with scripted models, no network and no keys. The first runs the scenario above, the worker death included, then checks the planner's generated actions: each awaits its result in the space Bridge.RESULT_SPACES names, each carries a description, and the assets action returns its telemetry. The second staffs the console (no simulated officer) and checks the gate: a rejected remediation closes as rejected, the console refuses the approval with the reason and audits it, a stray approval and the planner's dispatcher action write no order, an approved remediation waits for the person and then runs once, and the Incidents panel shows each incident's stage throughout. On 2026-11-04 mvn test took 41.5 s in all, 40.5 s of it the two fleet tests, with all seven tests passing.
// Act one: the incident lands, triage-a takes it, and we kill triage-a. OpsCenter.raise(ops.console(), "INC-1", "db-primary", "disk usage alarm: data volume at 98 percent", "high"); assertThat(ops.crash().awaitTake(Duration.ofSeconds(30))).as("a worker took the incident").isTrue(); String victim = ops.crash().victim(); String survivor = victim.equals("triage-a") ? "triage-b" : "triage-a"; ops.crash().kill(); ops.peer(victim).close(); Diagnosis diagnosis = OpsCenter.await(() -> OpsCenter.first(evidence, Diagnosis.class, "INC-1"), Duration.ofSeconds(120)).orElseThrow(); assertThat(diagnosis.diagnosedBy()).as("the surviving worker finished the job").isEqualTo(survivor); assertThat(diagnosis.cause()).contains("db-primary");
// Act two: the proposal is irreversible, the change-control judge dissents, the quorum approves. RemediationProposal proposal = OpsCenter.await( () -> OpsCenter.first(evidence, RemediationProposal.class, "INC-1"), Duration.ofSeconds(60)) .orElseThrow(); assertThat(proposal.reversible()).isFalse(); RemediationDecision decision = OpsCenter.await( () -> OpsCenter.first(evidence, RemediationDecision.class, "INC-1"), Duration.ofSeconds(60)) .orElseThrow(); assertThat(decision.outcome()).isEqualTo("approve"); assertThat(decision.tally()).contains("approve=2").contains("reject=1"); List<RiskReview> reviews = OpsCenter.await(() -> { List<RiskReview> found = OpsCenter.readAll(evidence, RiskReview.class, "INC-1"); return found.size() >= 3 ? Optional.of(found) : Optional.<List<RiskReview>>empty(); }, Duration.ofSeconds(30)).orElseThrow(); assertThat(reviews).extracting(RiskReview::judge) .containsExactlyInAnyOrder("judge-security", "judge-change", "judge-ops"); assertThat(reviews).filteredOn(r -> r.judge().equals("judge-change")) .extracting(RiskReview::option).containsExactly("reject");
// Act three: the officer approves, exactly one executor runs it, the recorder closes it. OperatorApproval approval = OpsCenter.await( () -> OpsCenter.first(evidence, OperatorApproval.class, "INC-1"), Duration.ofSeconds(60)) .orElseThrow(); assertThat(approval.operator()).isEqualTo("watch-officer"); IncidentRecord record = OpsCenter.await(() -> OpsCenter.first(evidence, IncidentRecord.class, "INC-1"), Duration.ofSeconds(120)).orElseThrow(); assertThat(record.outcome()).isEqualTo("remediated"); assertThat(record.evidence()).anySatisfy(line -> assertThat(line).startsWith("Telemetry")); assertThat(record.evidence()).anySatisfy(line -> assertThat(line).contains("judge-change voted reject")); assertThat(record.evidence()).anySatisfy(line -> assertThat(line).startsWith("Approval")); assertThat(record.evidence()).anySatisfy(line -> assertThat(line).startsWith("Receipt")); assertThat(record.evidence()).allSatisfy(line -> assertThat(line).contains(" · by ")); Thread.sleep(2000); // let any duplicate surface before counting assertThat(OpsCenter.readAll(evidence, RemediationReceipt.class, "INC-1")) .as("the ordered log commits exactly one executor").hasSize(1);
// The routine path: reversible, unanimous, still exactly once. OpsCenter.raise(ops.console(), "INC-2", "checkout-api", "p99 latency over 2 seconds", "medium"); RemediationDecision routine = OpsCenter.await( () -> OpsCenter.first(evidence, RemediationDecision.class, "INC-2"), Duration.ofSeconds(120)) .orElseThrow(); assertThat(routine.outcome()).isEqualTo("approve"); assertThat(routine.tally()).contains("approve=3").doesNotContain("reject=1"); OpsCenter.await(() -> OpsCenter.first(evidence, IncidentRecord.class, "INC-2"), Duration.ofSeconds(120)) .orElseThrow(); Thread.sleep(2000); assertThat(OpsCenter.readAll(evidence, RemediationReceipt.class, "INC-2")).hasSize(1); // The planner seat: the fleet's cards become invocable actions, and a generated agent. RemoteActions remote = Bridge.remoteActions(ops.planner()); RemoteAction triage = Bridge.awaitTriage(remote, Duration.ofSeconds(30)).orElseThrow(); Optional<Diagnosis> remoteDiagnosis = triage.invoke( new Incident("INC-3", "auth-gateway", "error rate above 10 percent", "high"), Diagnosis.class, Duration.ofSeconds(90)); assertThat(remoteDiagnosis).isPresent(); assertThat(remoteDiagnosis.orElseThrow().incidentId()).isEqualTo("INC-3"); EmbabelRemoteActions bridge = Bridge.bridge(remote); Object fleetAgent = bridge.agentInstance().orElseThrow(); assertThat(Arrays.stream(fleetAgent.getClass().getMethods()).map(Method::getName)) .contains("triage_Incident");
// Every generated action awaits its result in the space its agent writes to, and says what it does. Map<String, String> resultSpaces = Map.of( "triage_Incident", OpsFleet.EVIDENCE, "assets_Incident", OpsFleet.EVIDENCE, "remediation_Diagnosis", OpsFleet.EVIDENCE, "dispatcher_OperatorApproval", OpsFleet.ACTIONS, "watch-officer_RemediationDecision", OpsFleet.EVIDENCE, "recorder_RemediationReceipt", OpsFleet.EVIDENCE, "recorder_RemediationDecision", OpsFleet.EVIDENCE, "executor_RemediationOrder", OpsFleet.EVIDENCE); Supplier<Set<String>> names = () -> remote.available().stream().map(RemoteAction::name).collect(Collectors.toSet()); Set<String> generated = OpsCenter.await( () -> Optional.of(names.get()).filter(n -> n.containsAll(resultSpaces.keySet())), Duration.ofSeconds(30)).orElseGet(names); assertThat(generated).as("an action with no entry in the table above would wait in the wrong space") .isEqualTo(resultSpaces.keySet()); for (RemoteAction action : remote.available()) { assertThat(action.resultSpace()).as(action.name()) .isSameAs(ops.planner().space(resultSpaces.get(action.name()))); assertThat(action.declared().orElseThrow().description()).as(action.name() + " description") .isNotBlank(); } RemoteAction assets = remote.consuming(Incident.class).stream() .filter(a -> a.name().equals("assets_Incident")).findFirst().orElseThrow(); assertThat(assets.invoke(new Incident("INC-5", "checkout-api", "p99 latency over 2 seconds", "medium"), Telemetry.class, Duration.ofSeconds(30))) .as("the assets action finds its Telemetry in evidence, not in incidents").isPresent();
// The quorum rejects the irreversible credential purge, and the incident closes as rejected. RemediationDecision rejection = OpsCenter.await( () -> OpsCenter.first(evidence, RemediationDecision.class, "INC-4"), Duration.ofSeconds(60)).orElseThrow(); assertThat(rejection.outcome()).isEqualTo("reject"); assertThat(rejection.tally()).contains("approve=1").contains("reject=2"); IncidentRecord rejected = OpsCenter.await(() -> OpsCenter.first(evidence, IncidentRecord.class, "INC-4"), Duration.ofSeconds(30)).orElseThrow(); assertThat(rejected.outcome()).isEqualTo("rejected"); assertThat(rejected.evidence()).anySatisfy(line -> assertThat(line).startsWith("Decision · reject")); assertThat(rejected.evidence()).filteredOn(line -> line.startsWith("Review")).hasSize(3); assertThat(rejected.evidence()).noneSatisfy(line -> assertThat(line).startsWith("Approval")); assertThat(rejected.evidence()).noneSatisfy(line -> assertThat(line).startsWith("Receipt")); // The console refuses the approval, says why, and audits the refusal. Console.Reply refused = console.approve("INC-4", "approve it anyway"); assertThat(refused.status()).isEqualTo(400); assertThat(refused.body()).contains("\"ok\":false").contains("the risk panel rejected INC-4").contains("reject=2"); assertThat(console.approve("INC-99", "no such incident").body()).contains("has not decided on INC-99"); assertThat(ops.console().space(OpsFleet.AUDIT).readAll(Template.of(FleetCommander.C2Record.class), 50)) .anySatisfy(record -> { assertThat(record.action()).isEqualTo("approve-remediation refused"); assertThat(record.detail()).contains("INC-4"); }); // An approval written around the console reaches the dispatcher, which refuses it too, whoever wrote it. evidence.write(new OperatorApproval("INC-4", Remediation.proposalId("INC-4"), "stray", "written around the console"), Lease.of(Duration.ofHours(1))); RemoteActions remote = Bridge.remoteActions(ops.planner()); RemoteAction dispatcher = OpsCenter.await(() -> remote.consuming(OperatorApproval.class).stream() .filter(a -> a.name().equals("dispatcher_OperatorApproval")).findFirst(), Duration.ofSeconds(30)).orElseThrow(); assertThat(dispatcher.invoke(new OperatorApproval("INC-4", Remediation.proposalId("INC-4"), "planner", "through the dispatcher action"), RemediationOrder.class, Duration.ofSeconds(4))) .as("the dispatcher answers a rejected incident with no order").isEmpty(); assertThat(OpsCenter.readAll(actions, RemediationOrder.class, "INC-4")).isEmpty(); assertThat(OpsCenter.readAll(evidence, RemediationReceipt.class, "INC-4")).isEmpty();
// A staffed center waits: the panel approves INC-6 and nothing runs until a person approves. RemediationDecision approved = OpsCenter.await( () -> OpsCenter.first(evidence, RemediationDecision.class, "INC-6"), Duration.ofSeconds(60)).orElseThrow(); assertThat(approved.outcome()).isEqualTo("approve"); String decided = "decided: approve (" + approved.tally() + ")"; assertThat(OpsCenter.await(() -> console.stage("INC-6").filter(decided::equals), Duration.ofSeconds(10))) .as("the Incidents panel still shows INC-6 after triage took it off the board").isPresent(); assertThat(console.row("INC-6")).containsEntry("source", "checkout-api").containsEntry("severity", "medium"); Thread.sleep(1500); assertThat(OpsCenter.readAll(evidence, OperatorApproval.class, "INC-6")).isEmpty(); assertThat(console.approve("INC-6", "approved from the console").status()).isEqualTo(200); IncidentRecord record = OpsCenter.await(() -> OpsCenter.first(evidence, IncidentRecord.class, "INC-6"), Duration.ofSeconds(60)).orElseThrow(); assertThat(record.outcome()).isEqualTo("remediated"); assertThat(record.evidence()).anySatisfy(line -> assertThat(line).startsWith("Approval · operator:"));
The limits to keep in mind
- Delivery is at least once outside the actions space, so a diagnosis can in rare cases be produced twice. It is idempotent.
- Reads come from the local replica, so a peer sees an entry after replication reaches it.
- A quorum proves three judges agreed, and nothing more. The simulated officer approves whatever the panel approved; a staffed center sets
OPSCENTER_OPERATOR=consoleand waits for a person. The panel binds the person through the console command and the dispatcher; the actions space itself accepts an order from any peer in this open group. - The executors' Raft logs live in memory (
MemoryLogStore), so a restarted executor rejoins with an empty log.OrderedTakes.over(..., store)takes aFileLogStoreper seat when the log has to survive a restart. - The
fleet-controlspace exists for the console's "Broadcast directive" form. No agent attaches aDirectiveGate, so nothing honors a directive. - The scripted models are rules. They keep the demo and the test deterministic and keyless; a live model changes the diagnosis text and can change a vote.
- Evidence lines are
PEER_ASSERTED, so they name the hosting peer. Per-agent keys (PeerIdentity.subordinate) would make themAGENT_ATTESTEDand name the agent.
Where to go next
- The developer guide explains each primitive used here:
@SpaceTake,@SpaceNotify,@OrderedTake, the vote capability, the ordered log, Spring AI, Embabel and the console. - The workflow guide names the patterns this app combines: exactly once, deciding by vote, and a person in the loop.
- The Party Bus guide builds a larger fleet on the same choices: peers for votes and clerks, a planner bridge and a console.
- The Superfriends guide shows a fleet of services from several vendors answering one question.
- The architecture guide covers the five layers this page names below.