aspace://guide/opscenter

Ops Center

An incident lands on a shared board and an operations center works it. Specialists claim the parts they understand, one triage worker dies mid-job and its task comes back, three judges disagree and a quorum decides, a person approves from the fleet console what the panel approved, the irreversible step runs exactly once through a Raft ordered log, and the closed incident carries every piece of evidence with the peer that signed it. Twelve peers run in one JVM with Spring AI models that need no key, and under a profile an Embabel planner reaches the fleet through its cards. Every code window comes from the project's sources.

opscenter 0.5.0AgentSpaces 0.5.0SPEC v0.1.24Java 2112 peers · 14 agents · 6 spacesSpring AI 2.0.1Embabel 1.5.2 under -Pembabelone JVM, loopback TCPno API keys neededApache 2.0
The life of an incidentWhen a worker diesThe plannerRun it
aspace://guide/opscenter/idea

The idea: an operations center that runs itself

Ops Center is the Autonomous Operations Center flagship of the AgentSpaces pattern catalog, written as a standalone Java project. An incident lands on a shared board. Specialist agents claim the parts they understand and write what they learn beside the incident. One triage worker dies in the middle of its job and the incident comes back to the other. Three judges, each on its own peer, disagree about the proposed remediation and a quorum decides. A person approves it from the fleet console, and only if the quorum approved it: a remediation the quorum rejected closes as rejected and never reaches an executor. The irreversible step runs once, because three executors race for the order through a Raft log. The closed incident carries every piece of evidence, and each line names the peer that signed it.

No agent calls another and nothing coordinates them from the middle. Each agent reacts to a record it understands in one of four working spaces and writes the next record, so the whole lifecycle is a chain of typed entries. Read this guide beside the developer guide, which explains each primitive, and the workflow guide, which names the patterns; this page shows one application that uses six of them together.

consoletriage-a, triage-bstaffrisk panelexec-a, -b, -cIncidentwritten · incidents1Telemetry@SpaceNotify · evidence2Diagnosis@SpaceTake 15s · evidence2RemediationProposal@SpaceNotify · evidence3vote opened@Propose · reviews43 signed ballots@Ballot · reviews5RemediationDecision@OnDecision · evidence6OperatorApprovalthe human gate · evidence7RemediationOrder@SpaceNotify · actions8RemediationReceipt@OrderedTake · evidence9IncidentRecord@SpaceNotify · evidence10a worker dies holding the take; its 15 s lease lapsesand the other triage peer takes the incident againjudge-change dissents on an irreversible change:the tally is approve=2, reject=1a person signs from theconsole, or the demo'ssimulated officer doesif the panel rejects, the recorder closes theincident as rejected: no approval, no orderthree executors race for the order; the Raft logcommits one claim, so one receipt is writtenevery entry about the incident is read withits signer, and the record lists them
The life of one incident. One incident passes through ten steps in four spaces. Nine kinds of agent on five groups of peers take part, and no agent calls another. Each agent reacts to the record it understands and writes the next: telemetry and a diagnosis come from the incident, a proposal from the diagnosis, a vote from the proposal, an approval from the decision, an order from the approval, a receipt from the order, and the closed record from the receipt. A decision of reject skips steps 7 to 10: the recorder closes the incident as rejected. The four shaded steps are the stories the next chapters tell.

The scenario has two incidents. INC-1 is the hard one: the disk on db-primary fills, the first triage worker dies on the job, the proposed fail-over is irreversible, and the change-control judge dissents. INC-2 is the routine path: checkout latency, a reversible rollback, a unanimous panel. Both end with exactly one receipt. The demo, the test and this guide all run both. A third scripted incident exists for the other outcome. payments-vault reports a service account key exposed in a public repository, the proposed purge is irreversible and touches a credential, so two judges reject it and the quorum rejects it. The console session and the second fleet test raise it, and it closes as rejected with no approval and no receipt.

The story beat What carries it Where in the code
an incident landsa typed record in the incidents space; a monitor, a person or a planner can write oneOpsCenter.raise, the console's "Raise incident"
specialists claim parts@SpaceTake under a 15 s lease for triage, @SpaceNotify reactions for the restTriage, Assets, Remediation
a worker dies and its task returnsthe take lease lapses and the entry reappearsOpsCenter.CrashSwitch
three judges disagree and a quorum decides@Propose, @Ballot, @OnDecision on the vote capability, one voter per peerRiskPanel
a person approves what the panel approvedan OperatorApproval from the console or the simulated officer; the console and the dispatcher both refuse one for a rejected remediationWatchtower, Operations.WatchOfficer, RiskPanel.refusal
a rejected remediation closes as rejectedthe recorder's second cue, the RemediationDecision, writes an IncidentRecord with outcome rejectedOperations.Recorder.closeRejected
the irreversible step runs once@OrderedTake over a three-member Raft logOperations.Executor, OpsFleet.formExecutionQuorum
every piece of evidence, each with its signerreadAllIssued returns each entry with its issuerOperations.Recorder
an operator watches the stagesthe console's Incidents panel, one row per incidentWatchtower
a planner reaches the fleetRemoteActions and the Embabel bridge turn cards into typed actionsBridge, OpsCenterEmbabelApplication

It runs on a laptop with no API keys. Without OPENAI_API_KEY the agents' models answer from a script, through the same Spring AI ChatClient path as a live model, format instructions and structured output included. With the key set, the same clients call OpenAI. The one profile that needs more is -Pembabel, which adds the Embabel planner.

aspace://guide/opscenter/cast · chapter 1

The cast: twelve peers and fourteen agents

The center has 12 peers and 14 agents. Nine agent classes make the fourteen, because Triage, Judge and Executor run on several peers each. Every peer is a separate PeerNode with its own key, port and replicas, so closing one is what the fleet sees when a process dies. Two peers sit apart from the rest: the console hosts the human's seat, and the planner hosts no agent at all.

ONE GROUP opscenter: 12 PEERS, 14 AGENTS, 6 SPACES ON EVERY PEER, ONE JVM, LOOPBACK TCPTHE OPERATOR'S SEAT (the seed)console :7800WatchOfficer, fleet console :7591TRIAGE WORKERS (take under lease)triage-aTriage :7801triage-bTriage :7802STAFF (choreography)staff :7803Assets, Remediation, Dispatcher, RecorderRISK PANEL (a quorum counts one ballot per peer)risk-leadLead :7804judge-securityJudge :7805judge-changeJudge :7806judge-opsJudge :7807EXECUTION QUORUM (Raft)exec-a:7808exec-b:7809exec-c:7810THE PLANNER SEATplanner :7811no agentsRemoteActions over the fleet's cards; under -Pembabel a generated Embabel agentevery other peer seeds from console, :7800the console peer also serves the fleet console over HTTP on :7591
Twelve peers by role. Each box is a separate PeerNode with its own key, its own TCP port and its own replica of every space, and its second line names the agents it hosts. The planner hosts none: it reads the fleet's cards. Closing a peer is the event the rest of the fleet sees when a process dies.
Agent class (card name) Peer Annotation Reads, then writes
Triage (triage)triage-a, triage-b@SpaceTake on incidents, lease 15 sIncident (and the Telemetry on record), then Diagnosis in evidence
Assets (assets)staff@SpaceNotify on incidentsIncident, then Telemetry in evidence
Remediation (remediation)staff@SpaceNotify on evidenceDiagnosis, then RemediationProposal
Operations.Dispatcher (dispatcher)staff@SpaceNotify on evidenceOperatorApproval, the panel's RemediationDecision and the proposal, then RemediationOrder in actions; no order when the panel did not approve
Operations.Recorder (recorder)staff2 x @SpaceNotify on evidenceRemediationReceipt (or a rejecting RemediationDecision) and every entry about the incident, then IncidentRecord, outcome remediated or rejected
RiskPanel.Lead (risk-lead)risk-lead@Propose, @OnDecisionRemediationProposal, then a vote Proposal; the closed vote, then RemediationDecision
RiskPanel.Judge (risk-judge)judge-security, judge-change, judge-ops@Ballot on reviewsthe vote Proposal, then a signed ballot and a RiskReview
Operations.Executor (executor)exec-a, exec-b, exec-c@OrderedTake on actions, lease 30 sRemediationOrder, then RemediationReceipt in evidence
Operations.WatchOfficer (watch-officer)console@SpaceNotify on evidenceRemediationDecision, then OperatorApproval (headless only, and only for approve)

Every @SpaceTake, @SpaceNotify, @OrderedTake and @Propose method also sets a description, which the card publishes as that action's description, so a planner that lists the fleet's actions reads what each one does. Two components are not agents. Watchtower is the fleet console, served from the console peer on port 7591, and Bridge gives the planner peer its view of the fleet. The method that boards all of this is OpsCenter.board: it starts the twelve peers in order, binds each agent on its peer, and forms the execution quorum.

OpsCenter.board
/** Builds the fleet and binds every agent. Port 0 for the console picks a free one. */
public static Seating board(Models models, Options options) throws Exception {
    int base = options.basePort();
    List<Peer> peers = new ArrayList<>();
    Peer console = seat(peers, "console", base, 0);
    Peer triageA = seat(peers, "triage-a", base + 1, base);
    Peer triageB = seat(peers, "triage-b", base + 2, base);
    Peer staff = seat(peers, "staff", base + 3, base);
    Peer riskLead = seat(peers, "risk-lead", base + 4, base);
    Peer judgeSecurity = seat(peers, "judge-security", base + 5, base);
    Peer judgeChange = seat(peers, "judge-change", base + 6, base);
    Peer judgeOps = seat(peers, "judge-ops", base + 7, base);
    Peer execA = seat(peers, "exec-a", base + 8, base);
    Peer execB = seat(peers, "exec-b", base + 9, base);
    Peer execC = seat(peers, "exec-c", base + 10, base);
    Peer planner = seat(peers, "planner", base + 11, base);

    CrashSwitch crash = new CrashSwitch();
    Triage.Hook hook = options.stageCrash() ? crash : Triage.Hook.NONE;
    triageA.bind(new Triage("triage-a", models.worker("triage"), hook));
    triageB.bind(new Triage("triage-b", models.worker("triage"), hook));

    staff.bind(new Assets("assets"));
    staff.bind(new Remediation("remediation", models.chat("remediation")));
    staff.bind(new Operations.Dispatcher());
    staff.bind(new Operations.Recorder());

    riskLead.bind(new RiskPanel.Lead());
    judgeSecurity.bind(new RiskPanel.Judge("judge-security", "security: trust boundaries and credentials",
            models.chat("judge-security")));
    judgeChange.bind(new RiskPanel.Judge("judge-change", "change control: reversibility and maintenance windows",
            models.chat("judge-change")));
    judgeOps.bind(new RiskPanel.Judge("judge-ops", "operations: customer impact and time to recover",
            models.chat("judge-ops")));

    for (OpsFleet.Seat seat : OpsFleet.formExecutionQuorum(List.of(execA, execB, execC))) {
        seat.peer().bind(new Operations.Executor(seat.raft(), seat.peer().name()));
    }

    console.bind(new Operations.WatchOfficer("watch-officer", options.simulatedOperator(), options.reviewTime()));
    Watchtower watchtower = console.run(Watchtower.over(console));
    int consolePort = watchtower.start(options.consolePort());

    return new Seating(List.copyOf(peers), console, planner, crash, watchtower, consolePort);
}
✓ twelve seats, fourteen binds: the executors bind in a loop over the quorum's seats
aspace://guide/opscenter/fleet · chapter 2

How the fleet forms and finds itself

All twelve peers join one group named opscenter, whose founding is the text opscenter-v1. The group id comes from that text, the membership policy is OPEN, and the default strategy is LEASE_RACE. The first peer, console, listens on port 7800 with no seed; every other peer dials it on 127.0.0.1:7800 over plain TCP. Ports run from 7800 to 7811 in the order the peers start. Every frame carries a signature (the node's default ChannelAuth.SIGNED) and there is no TLS, which suits a demo on loopback. The library describes an OPEN group as Sybil-exposed by construction, so a fleet that must refuse strangers uses an invitation or a policy validator instead.

HOW A PEER JOINS: group opscenter, founding the text "opscenter-v1"triage-aport 7801consolethe first peer, port 7800the other tenports 7802 to 78111 dial the seed127.0.0.1:7800 over tcp2 introducesigned peer ad, group id checked3 membershipthe newcomer is introduced to the members4 cards and adsagents, spaces, capabilities on one gossip stream5 spaces catch updigests, then rumorsmembership policy OPEN, every frame signed (ChannelAuth.SIGNED, no TLS on loopback)SWIM probes every 1 s, a member silent for 30 s leaves the view; seeds are plain tcp endpoints
How a peer joins. A new peer needs only a seed to dial. It proves a key, learns the members, hears every card, and its spaces fill from the replicas already there. No peer in the middle admits it and no registry holds it: the group id is the hash of the founding, and membership is what the members agree on. Here the first peer, console, has no seed and the other eleven dial it.
Setting Value Where
group name, foundingopscenter, the text opscenter-v1OpsFleet.GROUP, FOUNDING
membership policy, default strategyOPEN, LEASE_RACEOpsFleet.group()
seedconsole at 127.0.0.1:7800; every other peer dials itOpsCenter.board
ports7800 to 7811, one per peer in start order; the console server on 7591Options.basePort, consolePort
transport, channelplain TcpTransport, every frame signed (ChannelAuth.SIGNED)OpsFleet.startPeer
membershipSWIM, member TTL 30 s, ping timeout 2 s, 2 indirect probesGroupMembership.Config
tick, settle window250 ms, 200 msOpsFleet.TICK, settleWindow

Membership runs on SWIM: each peer probes one member per second, waits two seconds for an answer before asking two others to probe, and drops a member that has been silent for 30 seconds. One clock drives all of it. Each peer starts ticking at 250 ms, and that tick advances membership, space replication, votes and the Raft log.

ONE PEER: OpsFleet.startPeer(name, port, seedPort)identityPeerIdentity.generate(): a fresh Ed25519 key; the peer id is its hashnodePeerNode.builder(identity).build(); listen on tcp 127.0.0.1:portgroupjoinGroup(group(), Config(30s, 2s, 2), seeds): SWIM, gossipdiscoveryDiscoveryService over an AdCache: cards and adsreplicassix ReplicatedSpace, settle window 200 ms, one per spacefacadeAgentSpaces.register(...).capabilities(new CapabilityRuntime(...))voteVoteCapability on reviews; group.provide(vote)tickstartTicking(250 ms): membership, gossip, votes, RaftTHEN, PER ROLEgroup.bind(agent)AgentBinder subscribes; the card goes outformExecutionQuorumthree seats, one member list,OrderedTakes.over(...), thengroup.ordered(actions, ordered)Watchtower.over(console)the fleet console on :7591Bridge.remoteActionsthe planner reads cards
Boarding a peer. Every peer runs the same eight steps. The fleet is built in code through the AgentSpaces facade because a vote needs a voter per peer and the ordered log needs one member list, both of which this method and formExecutionQuorum set up directly. One clock, the 250 ms tick, drives membership, replication, votes and the Raft log.
OpsFleet.startPeer
public static GroupAdvertisement group() {
    GroupId groupId = GroupId.fromFounding(FOUNDING.getBytes(StandardCharsets.UTF_8));
    return new GroupAdvertisement("aspace://" + groupId.value(), PeerIdentity.generate().peerId(), groupId,
            Instant.EPOCH, Duration.ofDays(365), GROUP, GroupAdvertisement.MembershipPolicy.OPEN,
            ConflictStrategyType.LEASE_RACE, GroupAdvertisement.GossipParameters.defaults());
}

How the agents find each other

Every bound agent publishes a signed AgentCard that names what it consumes and produces and which space each type lives in. Fourteen cards circulate here. The agent classes never read a card: the triage worker does not know the proposal writer exists, and each only watches the space and the record type it understands. The cards are for readers that plan, which in this app is the planner peer. It turns the cards into typed actions and routes each call by entry type, so the space delivers the work whether or not a card has arrived.

AGENT CARDS (one per bound agent, 15 min TTL, refreshed)triagetriage-atriagetriage-bassetsstaffremediationstaffdispatcherstaffrecorderstaffrisk-leadrisk-leadrisk-judgejudge-securityrisk-judgejudge-changerisk-judgejudge-opsexecutorexec-aexecutorexec-bexecutorexec-cwatch-officerconsoleplanner (RemoteActions)reads the cataloguea carda hint, never a rosterIncident goes to incidents,Diagnosis comes back in evidence:the planner routes by type
Cards and the catalogue. Every discoverable thing is a signed, typed, time-limited card on one gossip stream. A planner or a console reads the catalogue to learn what exists. The design rule that follows is to route by entry type: a space and a record type deliver the work, and a discovered name only tells a reader where to look.
aspace://guide/opscenter/wiring · chapter 3

Why the fleet is wired in code

The Spring Boot starter binds @SpaceTake, @SpaceNotify, @Ballot, @OnDecision and @CapabilityRef beans and provides the vote and aggregate capabilities. Two things in this scenario it does not do. The ordered log runs over a fixed member set that the starter cannot infer, and a quorum needs one voter per peer, so the three judges have to be three peers. The fleet therefore boards through the AgentSpaces facade, the way the numbered examples do, with group.provide(vote) on every peer and group.ordered(space, coordinator) on the three executors. The Party Bus demo makes the same choice.

A QUORUM COUNTS ONE BALLOT PER PEERthree judges on one peerjudge-securityjudge-changejudge-opsthree ballots from one peer count once: below the quorum of 3three judges on three peers (this app)judge-securityVoteCapabilityjudge-changeVoteCapabilityjudge-opsVoteCapabilitythree ballots from three peers: counted 3, quorum reachedeach peer carries its own VoteCapability, because a ballot'svoter must be that peer's own agentTHE ORDERED LOG NEEDS ONE MEMBER LISTmembers = [exec-a, exec-b, exec-c]the same list handed to each seat before any log startsexec-aRaftLogexec-bRaftLogexec-cRaftLoga majority of the three commits each claimthe other nine peers hold no log: they write orders andread receipts through the replicated spacesthe ordered log runs over a member set the starter cannotinfer, and a quorum needs a voter per peer, so thefleet boards its peers in code
Why the fleet boards in code. Under the default PEER granularity a peer casts one counted ballot however many agents it hosts, so three judges need three peers. The Raft log runs over a fixed set of members that every member must be given identically, so the three executors name each other when they start. The facade builds both in a few lines, and the Party Bus demo makes the same choice.

The vote rule comes from how the capability counts. The tally reads each ballot with its issuer and counts one ballot per issuing peer under the default PEER granularity, which the demo prints as counted per PEER. A ballot also has to name its own writer as voter, or the capability discards it as forged, so each peer builds its own VoteCapability over identity.agent(name). The ordered log gets its member list in formExecutionQuorum, which hands the same three peer ids to every seat before any log starts.

OpsFleet.formExecutionQuorum
/**
 * Forms the execution quorum. Every member is given the same fixed member list, and each
 * registers the coordinator for the actions space, which is what lets {@code @OrderedTake}
 * bind on that peer. The caller binds the executor agent on each seat.
 *
 * <p>Each log keeps its term, vote and entries in memory ({@code MemoryLogStore}, the default), so a
 * restarted executor rejoins with an empty log. {@code OrderedTakes.over(..., store)} takes a
 * {@code FileLogStore} per seat when the log must survive a restart; this demo does not, because
 * its executors live exactly as long as the JVM.
 */
public static List<Seat> formExecutionQuorum(List<Peer> executors) {
    List<PeerId> members = executors.stream().map(Peer::peerId).toList();
    List<Seat> seats = new ArrayList<>();
    long seed = 9000L;
    for (Peer executor : executors) {
        OrderedTakes ordered = OrderedTakes.over(executor.group().pipes(), executor.codec(), executor.identity(),
                executor.name(), members, InstantSource.system(), seed++, executor.space(ACTIONS));
        executor.group().ordered(ACTIONS, ordered);     // also advertises the Raft log
        seats.add(new Seat(executor, ordered.raft(), ordered));
    }
    return seats;
}
Where the wiring lives

Only OpsFleet and OpsCenter.board know about peers, ports, seeds and member lists. The nine agent classes carry annotations and no wiring code.

aspace://guide/opscenter/spaces · chapter 4

The six spaces

Every peer holds a replica of all six spaces. Four carry the work: incidents holds what a monitor, a person or a planner raises, evidence holds everything the agents learn, decide and do, reviews holds the vote, and actions holds the orders. The last two, fleet-control and c2-audit, belong to the console: the first for its directives and the second for its audit trail. The console needs fleet-control: whenever command-and-control is on, it shows a "Broadcast directive" form (PAUSE, RESUME, DRAIN) that writes a Directive there, and without the space the form answers "no control space". No agent in this app attaches a DirectiveGate, so a directive is written and audited and nothing honors it; a fleet that wants pause and drain from the console attaches a gate to each worker. Every space declares LEASE_RACE, the default. The executors take from the actions space through an OrderedTakes coordinator, which is how the library offers the ORDERED strategy, since ORDERED is not a setting on a space.

incidentsLEASE_RACEIncidentconsole and planner write;triage-a, triage-b taketake lease 15 sevidenceLEASE_RACETelemetryDiagnosisRemediationProposalRiskReviewRemediationDecisionOperatorApprovalRemediationReceiptIncidentRecordevery agent writes;reactions subscribe,nothing takes12 h leasesreviewsLEASE_RACE + voteProposalBallotrisk-lead opens;three judges vote12 h leasesactionsORDERED (Raft)RemediationOrderdispatcher writes;exec-a, -b, -c taketake lease 30 sfleet-controlLEASE_RACEDirectivethe console's Broadcastdirective form; nothing honors itc2-auditLEASE_RACEC2Recordthe console's audit trail
Six spaces, every one on every peer. Each of the twelve peers holds a replica of all six. Every space declares LEASE_RACE, the library's default; the executors take from the actions space through an OrderedTakes coordinator instead, which is how the library offers the ORDERED strategy (it is not a space setting). Nothing here auctions. The first four spaces carry the work; the last two belong to the console, which writes a directive to the first of them and an audit record for every command to the second.
Space and strategy Entry types Written by Read or taken by Lease
incidents
LEASE_RACE
Incidentconsole (12 h), planner (1 h)triage-a and triage-b take; Assets reactstake 15 s, entry 12 h
evidence
LEASE_RACE
Telemetry, Diagnosis, RemediationProposal, RiskReview, RemediationDecision, OperatorApproval, RemediationReceipt, IncidentRecordevery agent, and the console's approval commandreactions subscribe; the console and the recorder read12 h
reviews
LEASE_RACE plus the vote capability
vote Proposal, Ballotrisk-lead opens; the judges castthe tally on every peer12 h
actions
ORDERED through OrderedTakes
RemediationOrderdispatcherexec-a, exec-b, exec-c taketake 30 s, entry 12 h
fleet-control
LEASE_RACE
Directivethe console's Broadcast directive formnothing: no agent attaches a DirectiveGateset by the console
c2-audit
LEASE_RACE
C2Recordthe consoleanyone who auditsset by the console

Every record carries the incidentId it belongs to. That shared field ties a result to its request for the agents (their templates filter on it), for the recorder (the evidence chain is every entry issued about this incident), and for the Embabel bridge, which pairs a result with its request by matching shared record components. The records are plain Java records.

ONE ENTRY IN THE SPACE evidenceRemediationProposalincidentId INC-1proposalId prop-INC-1actionrationalereversible booleanproposedBy remediationspace strategyLEASE_RACE (never taken here)lease12 h resultLeasesigned bythe staff peer's key, PEER_ASSERTED
An entry. Every shared fact in an AgentSpaces app is a typed record in a space. The space's strategy says how a take is decided, the lease says how long the entry lives, tags make it findable, and the signature says which agent wrote it and lets any member check that.
Incidents: the records
/** What lands on the board. Written by a monitor, a person at the console, or a planner. */
public record Incident(String incidentId, String source, String summary, String severity) {
}
What to watch for

Leases are 12 hours on every entry except the take leases (15 s for triage, 30 s for the executors). A closed incident therefore stays readable in every replica for half a day, which is what lets the console and a late reader see the whole chain.

aspace://guide/opscenter/run · chapter 5

One run, end to end

OpsCenter.main picks the models from the environment, boards the fleet, waits 2.5 seconds for the cards to gossip, and runs the two incidents. For each one it raises the incident from the console peer and then waits on the evidence space for each stage in turn: diagnosis, proposal, decision, approval, receipt and the closed record, printing the elapsed time as each arrives. The demo reads only; every stage is written by the agent that owns it. If the decision is a rejection, runIncident skips to the closed record, because no approval or receipt will come.

# Who Cue Writes Code
1console, planner or a monitora person or a programIncidentOpsCenter.raise
2staff (Assets); triage-a or triage-bIncidentTelemetry; DiagnosisAssets.sample; Triage.diagnose
3staff (Remediation)DiagnosisRemediationProposalRemediation.propose
4risk-leadRemediationProposalvote Proposal remediation:INC-nLead.open
5judge-security, judge-change, judge-opsthe vote Proposalthree signed ballots; three RiskReviewsJudge.judge
6risk-leadthe closed voteRemediationDecisionLead.record
7console (WatchOfficer) or a persona RemediationDecision of approveOperatorApprovalWatchOfficer.review, Watchtower
7astaff (Recorder)a RemediationDecision of rejectIncidentRecord, outcome rejected; steps 8 to 10 do not happenRecorder.closeRejected
8staff (Dispatcher)OperatorApproval, once the panel's approve is on recordRemediationOrder, in actionsDispatcher.dispatch
9exec-a, exec-b or exec-cRemediationOrderRemediationReceiptExecutor.execute
10staff (Recorder)RemediationReceiptIncidentRecord, outcome remediatedRecorder.close
OpsCenter.runIncident
private static void runIncident(Seating ops, String id, String source, String summary, String severity,
                                boolean killFirstWorker) throws Exception {
    Space evidence = ops.console().space(OpsFleet.EVIDENCE);
    long start = System.nanoTime();
    raise(ops.console(), id, source, summary, severity);
    say(id + " written: " + summary);

    if (killFirstWorker) {
        if (ops.crash().awaitTake(Duration.ofSeconds(15))) {
            String victim = ops.crash().victim();
            say(victim + " took " + id + " and is consulting the model");
            ops.crash().kill();
            ops.peer(victim).close();
            say("kill -9 " + victim + ". Nothing renews its lease now; it lapses in 15 seconds.");
        } else {
            say("no worker took " + id + " within 15 seconds; both stay alive this run");
        }
    }

    Diagnosis diagnosis = await(() -> first(evidence, Diagnosis.class, id), Duration.ofSeconds(120))
            .orElseThrow(() -> new IllegalStateException("no diagnosis for " + id));
    say(elapsed(start) + " diagnosis by " + diagnosis.diagnosedBy() + ": " + diagnosis.cause());

    RemediationProposal proposal = await(() -> first(evidence, RemediationProposal.class, id), Duration.ofSeconds(60))
            .orElseThrow();
    say(elapsed(start) + " proposal: " + proposal.action() + " (reversible: " + proposal.reversible() + ")");

    RemediationDecision decision = await(() -> first(evidence, RemediationDecision.class, id), Duration.ofSeconds(60))
            .orElseThrow();
    say(elapsed(start) + " risk panel: " + decision.outcome() + " · tally " + decision.tally()
            + " · counted per " + decision.countedPer());

    if (!RiskPanel.APPROVE.equals(decision.outcome())) {
        // The panel binds the person: no approval is accepted and no order is placed. The recorder closes it.
        IncidentRecord rejected = await(() -> first(evidence, IncidentRecord.class, id), Duration.ofSeconds(60))
                .orElseThrow();
        say(elapsed(start) + " closed as " + rejected.outcome() + ". Evidence chain:");
        rejected.evidence().forEach(line -> say("    " + line));
        return;
    }

    OperatorApproval approval = await(() -> first(evidence, OperatorApproval.class, id), Duration.ofSeconds(600))
            .orElseThrow();
    say(elapsed(start) + " approved by " + approval.operator() + ": " + approval.note());

    RemediationReceipt receipt = await(() -> first(evidence, RemediationReceipt.class, id), Duration.ofSeconds(90))
            .orElseThrow();
    say(elapsed(start) + " executed once by " + receipt.executedBy() + " at log index " + receipt.commitIndex());

    IncidentRecord record = await(() -> first(evidence, IncidentRecord.class, id), Duration.ofSeconds(60))
            .orElseThrow();
    say(elapsed(start) + " closed. Evidence chain:");
    record.evidence().forEach(line -> say("    " + line));
}
THE RECORDED RUN OF 2026-11-04: SECONDS SINCE EACH INCIDENT WAS WRITTENtriageproposalrisk panelofficer's reviewexecutionINC-1db-primary16.617.919.221.321.5closed at 21.5 sthe dead worker's 15 s lease, then 1.2 s of model timeINC-2checkout-api1.52.84.15.96.4closed at 6.4 s0 s5 s10 s15 s20 s25 s
Where the seconds go. The first incident takes 21.5 s end to end and 16.6 s of that is the triage stage, which includes the dead worker's lease. The routine incident takes 6.4 s. After the diagnosis, the proposal, the panel, the officer's review and the execution take about the same time in both runs. The numbers are the ones the demo printed.
Recorded run, 2026-11-04
no OPENAI_API_KEY in the environment: the agents answer from the scripted models
models: scripted · operator: simulated · console: http://localhost:7591/ (token opscenter-operator)

== Incident 1: the disk fills on db-primary, and the first triage worker dies on the job ==
INC-1 written: disk usage alarm: data volume at 98 percent
triage-b took INC-1 and is consulting the model
kill -9 triage-b. Nothing renews its lease now; it lapses in 15 seconds.
Nov 04, 2026 3:00:02 AM ai.badmonkey.agentspaces.agent.AgentBinder lambda$startTakeLoop$0
WARNING: triage.diagnose: action failed; lease will lapse
java.lang.IllegalStateException: triage-b died mid-diagnosis
	...

[ 16.6s] diagnosis by triage-a: The data volume on db-primary is at 98 percent and the write-ahead log can no longer flush, so writes are stalling
[ 17.9s] proposal: Fail over to the standby and rebuild the primary's data volume (reversible: false)
[ 19.2s] risk panel: approve · tally approve=2, reject=1 · counted per PEER
[ 21.3s] approved by watch-officer: Simulated officer: approved after reading the tally (approve=2, reject=1)
[ 21.5s] executed once by exec-a at log index 3
[ 21.5s] closed. Evidence chain:
    Telemetry · db-primary disk.used.percent=98.4 % · by staff (PEER_ASSERTED)
    Diagnosis · The data volume on db-primary is at 98 percent and the write-ahead log can no longer flush, so writes are stalling (confidence 0.86) · by triage-a (PEER_ASSERTED)
    Proposal · Fail over to the standby and rebuild the primary's data volume · reversible: false · by staff (PEER_ASSERTED)
    Review · judge-ops voted approve: Customer impact outweighs the rebuild cost · by judge-ops (PEER_ASSERTED)
    Review · judge-change voted reject: An irreversible change outside a maintenance window needs a named owner first · by judge-change (PEER_ASSERTED)
    Review · judge-security voted approve: No change to the trust boundary · by judge-security (PEER_ASSERTED)
    Decision · approve · tally approve=2, reject=1 · counted per PEER · by risk-lead (PEER_ASSERTED)
    Approval · watch-officer: Simulated officer: approved after reading the tally (approve=2, reject=1) · by console (PEER_ASSERTED)
    Receipt · Fail over to the standby and rebuild the primary's data volume · executed by exec-a · log index 3 · by exec-a (PEER_ASSERTED)

== Incident 2: checkout latency, the routine path ==
INC-2 written: p99 latency over 2 seconds since the 14:10 deploy
[  1.5s] diagnosis by triage-a: The checkout API's connection pool is exhausted after the last deploy doubled its fan-out to the pricing service
[  2.8s] proposal: Roll back checkout-api to the previous release (reversible: true)
[  4.1s] risk panel: approve · tally approve=3, reject=0 · counted per PEER
[  5.9s] approved by watch-officer: Simulated officer: approved after reading the tally (approve=3, reject=0)
[  6.4s] executed once by exec-a at log index 6
[  6.4s] closed. Evidence chain:
    Telemetry · checkout-api p99.latency.ms=2350.0 ms · by staff (PEER_ASSERTED)
    Diagnosis · The checkout API's connection pool is exhausted after the last deploy doubled its fan-out to the pricing service (confidence 0.78) · by triage-a (PEER_ASSERTED)
    Proposal · Roll back checkout-api to the previous release · reversible: true · by staff (PEER_ASSERTED)
    Review · judge-change voted approve: Reversible, and the rollback path is known · by judge-change (PEER_ASSERTED)
    Review · judge-ops voted approve: Low blast radius · by judge-ops (PEER_ASSERTED)
    Review · judge-security voted approve: No change to the trust boundary · by judge-security (PEER_ASSERTED)
    Decision · approve · tally approve=3, reject=0 · counted per PEER · by risk-lead (PEER_ASSERTED)
    Approval · watch-officer: Simulated officer: approved after reading the tally (approve=3, reject=0) · by console (PEER_ASSERTED)
    Receipt · Roll back checkout-api to the previous release · executed by exec-a · log index 6 · by exec-a (PEER_ASSERTED)

INC-1: 1 receipt(s), exactly one expected
INC-2: 1 receipt(s), exactly one expected
✓ recorded 2026-11-04 on AgentSpaces 0.5.0 with scripted models; the SLF4J notices and the exception's stack frames are left out
What to watch for

The seconds vary a little from run to run. What does not vary is the shape: a 15 s gap in INC-1 where the dead worker's lease runs out, a dissent in the INC-1 tally, and one receipt per incident. The last two lines of the output count the receipts.

aspace://guide/opscenter/takes · chapter 6

Leased takes and the model call

Triage is the one agent that takes under the space's own lease race. The parameter type of diagnose becomes a take template on the incidents space, the binder performs the leased take, invokes the method, and when it returns, writes the Diagnosis to the evidence space and completes the take. An exception or a crash only lets the lease lapse, so the incident reappears. The method asks a Spring AI ChatClient for structured output with .entity(Reading.class); Spring AI appends the record's JSON schema to the prompt and binds the reply.

Triage.diagnose
@SpaceTake(space = OpsFleet.INCIDENTS, lease = "15s", pollTimeout = "300ms",
        resultSpace = OpsFleet.EVIDENCE, resultLease = "12h",
        description = "Diagnoses an incident: reads the telemetry on record, asks the model for the most "
                + "likely cause and a confidence, and writes the Diagnosis to the evidence space")
public Diagnosis diagnose(Incident incident) {
    hook.beforeThinking(name, incident);
    List<Telemetry> readings = evidence.readAll(
            Template.of(Telemetry.class).where("incidentId", eq(incident.incidentId())), 20);
    String telemetry = readings.isEmpty() ? "none yet" : readings.stream()
            .map(t -> t.asset() + " " + t.metric() + "=" + t.value() + " " + t.unit())
            .collect(Collectors.joining("; "));
    Reading reading = chat.prompt()
            .system(SYSTEM)
            .user("Incident " + incident.incidentId() + " from " + incident.source() + " (severity "
                    + incident.severity() + "): " + incident.summary() + "\nTelemetry: " + telemetry)
            .call()
            .entity(Reading.class);
    if (reading == null) {
        throw new IllegalStateException("the model returned no reading for " + incident.incidentId());
    }
    return new Diagnosis(incident.incidentId(), reading.cause(), reading.confidence(), name);
}
Triage.diagnose(Incident), INSIDE A @SpaceTake LEASE OF 15 sChatClientmodels.worker("triage")TakeLeaseAdvisorrenews the take per round tripOpenAiChatModelOPENAI_API_KEY set: gpt-4.1-miniScriptedChatModelno key: JSON per role, 1200 msJSON text in the model's reply.entity(Reading.class)Spring AI appends the record's JSON schema to the prompt (the format instructions) and binds the reply to Reading(cause, confidence)the method returns Diagnosis(incidentId, cause, confidence, diagnosedBy); the binder writes it to evidence and completes the takeONE CHATCLIENT PER ROLEtriageworker(): with the advisorremediationchat(): plain clientjudge-securitychat()judge-changechat()judge-opschat()only the take needs the advisor: a ballot or a reaction holds no lease to renew
A model call inside a lease. Every agent that thinks holds a Spring AI ChatClient built over either an OpenAI model or the scripted model, and both answer through the same structured-output path. The triage client carries TakeLeaseAdvisor, which renews the surrounding take on every model round trip, so a slow model does not let a healthy worker's lease lapse and a dead worker still loses the incident one lease after it stops.

Models builds one client per role. With OPENAI_API_KEY set, each role gets an OpenAiChatModel (model gpt-4.1-mini unless OPSCENTER_OPENAI_MODEL says otherwise, temperature 0.1). Without it, each role gets a ScriptedChatModel that answers in the JSON shape its agent expects after a think time of 1200 ms (OPSCENTER_THINK_MILLIS overrides it). The triage client alone carries the TakeLeaseAdvisor from agentspaces-springai, which renews the surrounding take on every model round trip. A slow model therefore keeps a healthy worker's claim, and a worker that hangs or dies stops renewing and loses it one lease later.

Models
public ChatClient chat(String role) {
    return ChatClient.builder(models.apply(role)).build();
}

public ChatClient worker(String role) {
    return ChatClient.builder(models.apply(role))
            .defaultAdvisors(new TakeLeaseAdvisor(scheduler, Duration.ofSeconds(30)))
            .build();
}

The other agents react. Assets and Remediation use @SpaceNotify, which subscribes to a record type without consuming it and writes the return value as the result. Remediation is a reaction rather than a take because a proposal is cheap to recompute, and every planner should see every diagnosis. The model decides whether the action is reversible, and that flag is what the risk panel argues about.

Reactions
@SpaceNotify(space = OpsFleet.INCIDENTS, lease = "12h", resultSpace = OpsFleet.EVIDENCE, resultLease = "12h",
        description = "Reports the current reading of the incident's asset as Telemetry in the evidence space")
public Telemetry sample(Incident incident) {
    Reading reading = READINGS.getOrDefault(incident.source(), new Reading("heartbeat", 1, "count"));
    return new Telemetry(incident.incidentId(), incident.source(), reading.metric(), reading.value(),
            reading.unit(), name);
}
What to watch for

Delivery is at least once outside the actions space. If a worker is slow rather than dead and a second worker also finishes, two diagnoses can appear. A diagnosis is idempotent, so the cost is a model call.

aspace://guide/opscenter/death · chapter 7

When a worker dies

The demo stages the failure with CrashSwitch, a hook both triage workers share. Whichever worker takes INC-1 first becomes the victim: it stalls in the hook before it consults the model, and the demo then calls kill() and closes that worker's peer. Closing a peer closes its agents, its facade, its replicas and its node, and the binder interrupts the worker thread, which is how the hook learns it has died. The hook throws, the take never completes, and nobody renews the lease.

INC-1 IN THE INCIDENTS SPACE, SECONDS AFTER IT WAS WRITTENincidentstriage-bthe demotriage-aconsoleheld by triage-b under a 15 s lease that nobody renewsheld by triage-athe peer closes, the hook throws, the take never completestakes the incident, thinks 1.2 sINC-1 writtenDiagnosis in evidence at 16.6 skill -9 triage-b0 s5 s10 s15 s20 s
A death costs one lease. The recorded diagnosis arrived at 16.6 s. The scripted model takes 1.2 s and the take lease is 15 s, which leaves 0.4 s before the first take for the write, the gossip and the first poll; the figure draws the first take at that point because the demo does not print it. The incident never left the replicated space, so no coordinator had to notice the death: the lease ran out and the other worker's next poll took the entry.
CrashSwitch
/**
 * Arranges the crash. Both triage workers share this hook. Whichever worker takes the
 * first incident becomes the victim: it stalls here, before the model is consulted, until
 * the demo closes its peer. Then the hook throws, so the take never completes, and the
 * lease lapses with nobody renewing it. Every other call, including the survivor's retry
 * of the same incident, passes straight through. If nobody kills the victim, the hook
 * gives up after a minute and the worker carries on.
 */
public static final class CrashSwitch implements Triage.Hook {
    private final CountDownLatch armed = new CountDownLatch(1);
    private volatile boolean killed;
    private volatile String victim;

    @Override
    public void beforeThinking(String worker, Incident incident) {
        synchronized (this) {
            if (victim != null) {
                return;
            }
            victim = worker;
        }
        armed.countDown();
        long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos();
        while (!killed && System.nanoTime() < deadline) {
            try {
                Thread.sleep(50);
            } catch (InterruptedException e) {
                // The binder interrupts its worker when the peer closes: that is the death.
                Thread.currentThread().interrupt();
                killed = true;
            }
        }
        if (killed) {
            throw new IllegalStateException(worker + " died mid-diagnosis");
        }
    }

    /** Waits until a worker is holding the first incident. */
    public boolean awaitTake(Duration timeout) throws InterruptedException {
        return armed.await(timeout.toMillis(), TimeUnit.MILLISECONDS);
    }

    /** The worker holding the first incident, once {@link #awaitTake} has returned true. */
    public String victim() {
        return victim;
    }

    public void kill() {
        killed = true;
    }
}

Fifteen seconds after the take, the lease lapses on every replica and the entry is takeable again. The other triage peer polls with a 300 ms timeout, takes it, passes straight through the hook, and finishes in one model call. The recorded run shows it: diagnosis by triage-a at 16.6 s, which is the lease plus the 1.2 s of model time and the time before the first take. The binder logs the dead worker's exception as a warning, triage.diagnose: action failed; lease will lapse, and that line is expected output.

What to watch for

The lease is the recovery time, so a shorter lease recovers faster and needs a worker that renews more often. Here the advisor does the renewing, once per model round trip. The test asserts the outcome that matters: after the kill, the diagnosis comes from the survivor, and the incident still closes.

aspace://guide/opscenter/quorum · chapter 8

Three judges and a quorum

Every remediation proposal goes to a risk panel. The lead opens a vote with one annotation, @Propose: the shape of the vote is fixed (options approve and reject, quorum 3, a 12 hour lease) and the cue decides only the question. The binder opens it once per incident, as the lead, and a redelivered proposal opens nothing. Each judge sits on its own peer, asks its own model from its own concern, writes a RiskReview with the reason, and returns an option, which the binder casts as a signed ballot. When three counted ballots are in, the lead's @OnDecision writes the RemediationDecision with the tally.

THE RISK PANEL: aspace:cap/vote ON THE reviews SPACERemediationProposalevidencerisk-lead@ProposeProposalremediation:INC-1options approve, reject; quorum 3; lease 12 hjudge-securitysecurityjudge-changechange controljudge-opsoperationstallyone counted ballot per peerties break by option namerisk-lead@OnDecisionRemediationDecisionoutcome, tally, countedPereach judge also writes a RiskReview with itsreason, beside its signed ballotWHO DISSENTS: THREE INCIDENTS (scripted models)judge-securityjudge-changejudge-opstallyINC-1 fail over, reversible: falseapproverejectapproveapprove=2, reject=1INC-2 roll back, reversible: trueapproveapproveapproveapprove=3, reject=0INC-4 purge key, reversible: falserejectrejectapproveapprove=1, reject=2the change-control judge rejects a proposal marked reversible: no; the security judge rejects one that touches a credential
Three judges, one decision. The lead opens one vote per remediation proposal with @Propose. Each judge reads the question, asks its own model, writes the reason and casts a signed ballot from its own peer; the arrows into the tally are those ballots. After three counted ballots the lead's @OnDecision records the outcome with a tally anyone can recount from the signed ballots, so the dissent in INC-1 stays on the record. INC-4 comes from the console session: two judges reject, so the decision is reject.
RiskPanel
/** Opens the vote for each proposal and writes the decision when it closes. */
@AgentSpec(name = "risk-lead", description = "Puts each remediation proposal to the risk panel and records the decision",
        goals = {"decide remediations"})
public static final class Lead {

    /**
     * One vote per remediation, as one annotation (ISSUE-Propose): the shape
     * is fixed and the cue decides only the question. The binder opens once
     * per incident, as this agent, and a redelivered proposal opens nothing.
     */
    @Propose(space = OpsFleet.EVIDENCE, vote = OpsFleet.REVIEWS, prefix = PREFIX, key = "incidentId",
            options = {"approve", "reject"}, quorum = QUORUM, lease = "12h", subscriptionLease = "12h",
            description = "Opens a risk-panel vote on a remediation proposal, one vote per incident")
    public String open(RemediationProposal proposal) {
        return "Approve remediation for " + proposal.incidentId() + ": " + proposal.action()
                + ". Reversible: " + (proposal.reversible() ? "yes" : "no")
                + ". Rationale: " + proposal.rationale();
    }

    @OnDecision(space = OpsFleet.REVIEWS, prefix = PREFIX, lease = "12h",
            resultSpace = OpsFleet.EVIDENCE, resultLease = "12h")
    public RemediationDecision record(VoteCapability.Decision decision) {
        String incidentId = incidentOf(decision.proposalId());
        String tally = new TreeMap<>(decision.tally()).entrySet().stream()
                .map(e -> e.getKey() + "=" + e.getValue())
                .collect(Collectors.joining(", "));
        return new RemediationDecision(incidentId, Remediation.proposalId(incidentId), decision.winner(), tally,
                decision.granularity().name());
    }
}

The scripted judges make INC-1 a genuine disagreement. The change-control judge rejects any proposal marked reversible: no, and the fail-over is irreversible, so the tally is two to one. The panel still approves. The dissent stays on the record twice: as a RiskReview with the reason, and as a signed ballot in the reviews space that anyone can recount. A third scripted incident goes the other way. For the key exposed in a public repository, the proposed purge is irreversible, so the change-control judge rejects it, and it touches a credential, so the security judge rejects it too. The operations judge approves, the tally is approve=1, reject=2, and the decision is reject. The next chapter shows what a rejection does.

What to watch for

A quorum proves that three judges agreed, and nothing more. The scripted judges are rules; live models can all be wrong together. The vote also counts peers, which is why the three judges are three peers, and a judge that returns null abstains rather than cast an option the proposal does not offer.

aspace://guide/opscenter/gate · chapter 9

The human gate

A remediation does not run on the panel's say alone, and it does not run on a person's say alone either. The dispatcher reacts to an OperatorApproval and to nothing else, and the gate lets a person approve only what the panel approved. Two paths write the approval. Headless, the WatchOfficer on the console peer approves a decision of approve after a 2 second review and says so in the note. With OPSCENTER_OPERATOR=console, the officer returns null and the run waits for a person to press "Approve remediation" in the console.

THE PANEL APPROVED (INC-1, INC-2): A PERSON OR THE SIMULATED OFFICER SIGNSRemediationDecisionoutcome approveWatchOfficerheadless, after 2 sApprove remediationthe console commandOperatorApprovalevidence, 12 h leaseDispatcherRiskPanel.refusal(): noneRemediationOrderactionsTHE PANEL REJECTED (INC-4): NO PATH WRITES AN ORDERRemediationDecisionoutcome rejectWatchOfficeranswers nothingApprove remediationrefused: HTTP 400closeRejectedRecorder, second cueno approval is writtenand c2-audit records the refusalIncidentRecordoutcome rejectedevery entry so far, with its signerAN APPROVAL WRITTEN AROUND THE CONSOLE FOR THE REJECTED INCIDENTOperatorApprovalfrom any other peerDispatcherRiskPanel.refusal()no order is written; the dispatcher logs the refusal and its reason
The panel binds the person. After an approve decision, either path writes the same OperatorApproval, and the dispatcher asks RiskPanel.refusal once more before it places the order. After a reject decision the officer answers nothing, the console refuses the command and audits the refusal, and the recorder closes the incident as rejected. An approval entry written around the console reaches the dispatcher, which refuses it for the same reason.

The panel binds the person in three places. The console command asks RiskPanel.refusal before it writes anything. For a rejected incident it answers HTTP 400 with the reason and the tally, for an incident the panel has not decided it says so, and in both cases it writes a c2-audit record of the refusal. The dispatcher asks the same question when an approval reaches it, because any peer can write an OperatorApproval entry and the console is only one of them. A decision that has not reached the staff peer's replica yet is waited for up to five seconds and then counts as no decision. When the panel rejected, the officer answers nothing and the recorder's second cue closes the incident as rejected.

The gate
/**
 * Why an operator approval of this incident must be refused, or empty when the panel passed it.
 * The human gate sits behind the panel: a person approves what the quorum approved and nothing
 * else, so a remediation the panel rejected never reaches the executors on a human's say. The
 * console command and the dispatcher both ask this, because an approval entry can also arrive
 * from a peer that is not the console. A decision that has not reached this replica yet is
 * waited for up to {@code wait}; a decision that is not there by then counts as no decision.
 */
public static Optional<String> refusal(Space evidence, String incidentId, Duration wait) {
    long deadline = System.nanoTime() + wait.toNanos();
    Optional<RemediationDecision> decision = decisionOn(evidence, incidentId);
    while (decision.isEmpty() && System.nanoTime() < deadline) {
        try {
            Thread.sleep(100);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            break;
        }
        decision = decisionOn(evidence, incidentId);
    }
    if (decision.isEmpty()) {
        return Optional.of("the risk panel has not decided on " + incidentId + " yet");
    }
    RemediationDecision decided = decision.get();
    return APPROVE.equals(decided.outcome()) ? Optional.empty()
            : Optional.of("the risk panel rejected " + incidentId + " (" + decided.tally() + ")");
}
What to watch for

The check lives in the console command and the dispatcher. It does not make the actions space refuse an order: any peer admitted to the group can write a RemediationOrder, as any peer can write any entry in this open demo group, and a deployment that must stop that restricts which peers may write that space. The approval itself is still one signed note that names an operator; the single static token audits as operator, and an OIDC validator would put each person's own subject in that field.

aspace://guide/opscenter/console · chapter 10

The console: stages and commands

The fleet console is the library's FleetConsoleServer, served from the console peer at http://localhost:7591/. Watchtower gives it the stock views (members, advertisements, the incidents, actions and evidence spaces with per-worker completion counts), an Incidents panel of its own, and two commands behind the operator token opscenter-operator. A command runs through FleetCommander under the console peer's key, so what it writes is a signed, leased entry, and the commander also writes a c2-audit record that names the command and the operator. With a single static token the library audits every holder as operator.

THE STAGE OF ONE INCIDENT: THE FURTHEST RECORD IN EVIDENCEopenthe Incident, kept in the rosterdiagnosedDiagnosis in evidenceunder reviewRemediationProposal in evidencedecided: approveRemediationDecision, outcome approveapprovedOperatorApproval in evidenceexecutedRemediationReceipt in evidenceclosed: remediatedIncidentRecord, outcome remediateddecided: rejectRemediationDecision, outcome rejectclosed: rejectedIncidentRecord, outcome rejectedthe recorder writes this one;no approval or receipt followsWHERE A ROW COMES FROMincidents spacethe Incident is gone once triage takes itrosternotify(WRITTEN) keeps each Incidentwrittenone row per incidentid, source, severity from the rosterevidence spacereadAll per record type gives the stagea panel that read only the incidents space would showa row until the first claim and nothing after it
The Incidents panel. A row's stage is the furthest record the evidence space holds for that incident, checked from the end of the lifecycle backwards, so a rejected remediation shows decided: reject and then closed: rejected. A triage worker's completed take removes the Incident from the board, so the console peer keeps a roster of every incident it saw written and reads the stage from the evidence space.

The panel answers one question per incident: how far has it got. It reads the stage from the evidence space, from the end of the lifecycle backwards, so the first record that exists names the stage. The incident itself needs more care. A triage worker's completed take removes the Incident from the board, and a row built only from the incidents space would exist until the first claim and then vanish. The panel therefore keeps a Roster: a notify subscription on the console peer's incidents space that remembers each incident written, which also supplies the source and the severity for the row. A rejected remediation reads decided: reject and then closed: rejected.

The Incidents panel
/**
 * What the console peer has seen written to the incidents space. A triage worker takes an
 * incident, and a completed take removes the entry, so the space alone shows an incident
 * only until the first worker claims it. The roster keeps each one for the panel, which
 * then reads the stage from the evidence space.
 */
private static final class Roster implements AutoCloseable {

    private static final int CAPACITY = 500;

    private final Space incidents;
    private final Map<String, Incident> seen = Collections.synchronizedMap(new LinkedHashMap<>() {
        @Override
        protected boolean removeEldestEntry(Map.Entry<String, Incident> eldest) {
            return size() > CAPACITY;
        }
    });
    private final Subscription subscription;

    Roster(Space incidents) {
        this.incidents = incidents;
        this.subscription = incidents.notify(Template.of(Incident.class), event -> {
            if (event.kind() == SpaceEvent.Kind.WRITTEN) {
                seen.putIfAbsent(event.entry().incidentId(), event.entry());
            }
        }, Lease.of(Duration.ofDays(365)));
    }

    /** Every incident this console has seen, oldest first, including any still unclaimed on the board. */
    List<Incident> all() {
        for (Incident incident : incidents.readAll(Template.of(Incident.class), 50)) {
            seen.putIfAbsent(incident.incidentId(), incident);
        }
        synchronized (seen) {
            return List.copyOf(seen.values());
        }
    }

    @Override
    public void close() {
        subscription.close();
    }
}

The two commands are "Raise incident", which writes an Incident the way a monitor would (its source list includes payments-vault), and "Approve remediation", the gate above. The page also carries a "Broadcast directive" form that the library adds whenever command-and-control is on. It writes a Directive into fleet-control, and nothing in this app honors it, because no agent attaches a DirectiveGate.

Recorded console session, 2026-11-04
[  0.0s] demo started
[  4.3s] panel  INC-1  db-primary     high      open
[  4.5s] POST raise-incident {"incidentId":"INC-4","source":"payments-vault","summary":"service account key exposed in a public repository","severity":"critical"} -> 200 {"ok":true,"message":"raised INC-4","entryId":"4684c6c6-77ab-45bc-9495-169f065c3ddf","operator":"operator"}
[  4.5s] panel  INC-4  payments-vault critical  open
[  6.1s] panel  INC-4  payments-vault critical  diagnosed
[  7.3s] panel  INC-4  payments-vault critical  under review
[  8.6s] panel  INC-4  payments-vault critical  closed: rejected
[  8.6s] POST approve-remediation {"incidentId":"INC-4","note":"approve it anyway"} -> 400 {"ok":false,"message":"approval refused: the risk panel rejected INC-4 (approve=1, reject=2)","operator":"operator"}
[  8.6s] POST approve-remediation {"incidentId":"INC-99","note":"no such incident"} -> 400 {"ok":false,"message":"approval refused: the risk panel has not decided on INC-99 yet","operator":"operator"}
[ 20.8s] panel  INC-1  db-primary     high      diagnosed
[ 21.9s] panel  INC-1  db-primary     high      under review
[ 23.2s] panel  INC-1  db-primary     high      decided: approve (approve=2, reject=1)
[ 23.3s] POST approve-remediation {"incidentId":"INC-1","note":"approved from the console after reading the tally"} -> 200 {"ok":true,"message":"approved INC-1","entryId":"febc96b4-3a01-4e6f-a6fb-49e24232818a","operator":"operator"}
[ 23.3s] POST directive PAUSE triage-a -> 200 {"ok":true,"message":"PAUSE sent to triage-a","operator":"operator"}
[ 23.4s] panel  INC-1  db-primary     high      approved
[ 23.7s] panel  INC-1  db-primary     high      closed: remediated
[ 23.8s] panel  INC-2  checkout-api   medium    open
[ 25.3s] panel  INC-2  checkout-api   medium    diagnosed
[ 26.5s] panel  INC-2  checkout-api   medium    under review
[ 27.8s] panel  INC-2  checkout-api   medium    decided: approve (approve=3, reject=0)
[ 27.8s] POST approve-remediation {"incidentId":"INC-2","note":"approved from the console after reading the tally"} -> 200 {"ok":true,"message":"approved INC-2","entryId":"f69286c5-71e4-45f6-a0f2-2c4229013b9b","operator":"operator"}
[ 27.9s] panel  INC-2  checkout-api   medium    approved
[ 28.3s] panel  INC-2  checkout-api   medium    closed: remediated
[ 28.3s] all three incidents closed
[ 30.6s] demo stopped
✓ recorded 2026-11-04 with OPSCENTER_OPERATOR=console (the script starts the demo as mvn exec:java -Dexec.args=--watch), polls the Incidents panel every 0.1 s and presses the commands over HTTP with the operator token; the lines are the script's own output

Read the session from the top. The script raised INC-4 through the console command at 4.5 s, while INC-1's first worker was dead and its lease had not yet lapsed, so the other triage worker took INC-4 and it reached closed: rejected at 8.6 s. The script then pressed Approve on it, and the console answered HTTP 400 with the reason and the tally. It also pressed Approve on INC-99, which the panel never decided, and got the matching 400. INC-1 reappeared when its lease lapsed, and the script approved it the moment the panel showed decided: approve, at 23.2 s. The PAUSE directive came back 200 and changed nothing, because no agent attaches a gate to fleet-control. The panel never listed executed, because the recorder closes the incident as soon as the receipt lands, so that stage falls between two of the script's 0.1 s polls.

What to watch for

The panel is a view over the console peer's replica, so a row can lag the fleet by a replication round. The stage names a record that exists, and it does not name who is working on the incident: while a worker holds an incident under its take lease the row still says open. The roster holds the last 500 incidents the console saw written; a record whose incident the console never saw still gets a row, with an empty source.

aspace://guide/opscenter/once · chapter 11

Exactly once: the executors

The dispatcher writes a RemediationOrder to the actions space, which every peer replicates. Three executor peers each run an @OrderedTake loop on it. The take goes through an OrderedTakes coordinator rather than the space's own claim race. Each executor that sees the order submits a signed claim as a Raft command, the log applies the first claim for the entry's epoch and rejects the rest, and the winner adopts the entry, runs execute, and completes with a receipt. The same decision falls out on every member because it depends only on the committed log.

RemediationOrderactions, on every peer@OrderedTakeexec-aRaft memberexec-bRaft memberexec-cRaft memberTHE RAFT LOGclaim 1: first appliedclaim 2: stale epochclaim 3: stale epochthe first committed claim forthe entry's epoch is adopted,identically on every memberwinner adopts the entryruns execute(), 300 ms of workRemediationReceiptexecutedBy, commitIndexrecorded run: INC-1 by exec-a atlog index 3, INC-2 by exec-a at 6;the index counts committedclaims and varies per runa member cut off from the other two cannot commit a claim, so it cannot take: exactly once costs liveness on a minority
The exactly-once desk. Each executor that sees the order submits a signed claim as a Raft command. The log applies the first and rejects the rest by epoch, identically on every member, so one executor adopts the take and writes the only receipt. The winner and the log index vary from run to run; the count of receipts does not, and the demo prints it.
Operations.Executor
/**
 * The one step that cannot happen twice. Three executors race for every order and the
 * Raft log commits exactly one claimant. The receipt carries the commit index, so the
 * record shows where in the log the action was committed.
 */
@AgentSpec(name = "executor", description = "Executes an approved remediation exactly once",
        goals = {"execute remediations"})
public static final class Executor {

    private final RaftLog raft;
    private final String name;

    public Executor(RaftLog raft, String name) {
        this.raft = raft;
        this.name = name;
    }

    @OrderedTake(space = OpsFleet.ACTIONS, lease = "30s", pollTimeout = "2s",
            resultSpace = OpsFleet.EVIDENCE, resultLease = "12h",
            description = "Executes an order exactly once: the Raft log commits one claimant, and the "
                    + "receipt carries the commit index")
    public RemediationReceipt execute(RemediationOrder order) {
        try {
            Thread.sleep(300);  // the work itself
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        return new RemediationReceipt(order.incidentId(), order.orderId(), order.action(), name,
                raft.commitIndex());
    }
}

The receipt records raft.commitIndex() at the moment the executor builds it. In the recorded run INC-1 reports log index 3 and INC-2 reports 6. Each executor that sees an order submits a claim command, and the index the receipt carries is the commit index at the moment the winner builds it, so it depends on how many claims had committed by then and differs from run to run. The winner was exec-a for INC-1 and exec-a for INC-2; it can vary, and the count of receipts does not. The demo prints the receipts and the test asserts one per incident, after a two second pause that gives any duplicate time to appear.

What to watch for

Exactly once costs liveness on a minority. A member that cannot reach a majority of the three cannot commit a claim, so it cannot take. Two of three executors must be up for any order to run. The log here lives in memory (the default MemoryLogStore), so a restarted executor rejoins with an empty log. OrderedTakes.over(..., store) takes a FileLogStore per seat, which keeps the term, the vote and the entries on disk, so an executor can rejoin with its vote intact after a restart. The demo keeps the default because its executors live exactly as long as the JVM. A remediation the panel rejected never reaches this chapter: no order is written, so no executor claims anything.

aspace://guide/opscenter/evidence · chapter 12

The evidence chain

The recorder reacts to the receipt and closes the incident. It assembles the record from what the space already holds: for each record type, it reads every entry about the incident with readAllIssued, which returns the entry together with its issuer and its attestation, and prints each line with the peer that signed it. It has two cues. A RemediationReceipt closes the incident as remediated. A RemediationDecision of reject closes it as rejected, with the same kinds of line up to the decision and no approval or receipt, after waiting up to three seconds for the three reviews to reach its replica, because the decision is written the moment the third ballot is counted.

Operations.Recorder
/**
 * Closes the incident. The record is assembled from what the space already holds: every
 * entry about the incident, read with its issuer, so each line names who signed it and
 * how strongly. Nothing here is narrated after the fact. Every line reads {@code PEER_ASSERTED}
 * because each peer signs with its own key and vouches for the agents it hosts; an agent
 * with a certified key of its own ({@code PeerIdentity.subordinate}) would read
 * {@code AGENT_ATTESTED} and name itself. An executed remediation closes as
 * {@code remediated}; one the risk panel rejected closes as {@code rejected}, with the
 * evidence gathered up to the decision and no approval or receipt in it.
 */
@AgentSpec(name = "recorder", description = "Closes an incident with its full evidence chain",
        goals = {"record incidents"})
public static final class Recorder {

    public static final String REMEDIATED = "remediated";
    public static final String REJECTED = "rejected";
    /** How long the judges' reviews may lag behind the decision on this peer's replica. */
    private static final Duration REVIEW_WAIT = Duration.ofSeconds(3);

    @SpaceRef(OpsFleet.EVIDENCE)
    Space evidence;

    @SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultLease = "12h",
            description = "Closes an executed incident as remediated, with the evidence chain: "
                    + "every entry about it, each line naming the peer that signed it")
    public IncidentRecord close(RemediationReceipt receipt) {
        return record(receipt.incidentId(), REMEDIATED);
    }

    @SpaceNotify(space = OpsFleet.EVIDENCE, lease = "12h", resultLease = "12h",
            description = "Closes an incident the risk panel rejected as rejected, with the evidence "
                    + "gathered up to the decision; writes nothing for an approved decision")
    public IncidentRecord closeRejected(RemediationDecision decision) {
        if (RiskPanel.APPROVE.equals(decision.outcome())) {
            return null;    // the receipt closes an approved incident
        }
        awaitReviews(decision.incidentId());
        return record(decision.incidentId(), REJECTED);
    }

    /** The decision is written once all ballots are in, and each review precedes its ballot; let the replica catch up. */
    private void awaitReviews(String incidentId) {
        long deadline = System.nanoTime() + REVIEW_WAIT.toNanos();
        while (reviews(incidentId) < RiskPanel.QUORUM && System.nanoTime() < deadline) {
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                return;
            }
        }
    }

    private int reviews(String incidentId) {
        return evidence.readAll(Template.of(RiskReview.class).where("incidentId", eq(incidentId)), 50).size();
    }

    private IncidentRecord record(String id, String outcome) {
        List<String> lines = new ArrayList<>();
        lines.addAll(issued(Telemetry.class, id, t -> "Telemetry · " + t.asset() + " " + t.metric() + "="
                + t.value() + " " + t.unit()));
        lines.addAll(issued(Diagnosis.class, id, d -> "Diagnosis · " + d.cause() + " (confidence "
                + d.confidence() + ")"));
        lines.addAll(issued(RemediationProposal.class, id, p -> "Proposal · " + p.action() + " · reversible: "
                + p.reversible()));
        lines.addAll(issued(RiskReview.class, id, r -> "Review · " + r.judge() + " voted " + r.option() + ": "
                + r.reason()));
        lines.addAll(issued(RemediationDecision.class, id, d -> "Decision · " + d.outcome() + " · tally "
                + d.tally() + " · counted per " + d.countedPer()));
        lines.addAll(issued(OperatorApproval.class, id, a -> "Approval · " + a.operator() + ": " + a.note()));
        lines.addAll(issued(RemediationReceipt.class, id, r -> "Receipt · " + r.action() + " · executed by "
                + r.executedBy() + " · log index " + r.commitIndex()));
        return new IncidentRecord(id, outcome, List.copyOf(lines));
    }

    private <T> List<String> issued(Class<T> type, String incidentId, Function<T, String> describe) {
        return evidence.readAllIssued(Template.of(type).where("incidentId", eq(incidentId)), 50).stream()
                .map(i -> describe.apply(i.entry()) + " · by " + i.issuer().localName() + " ("
                        + i.attestation() + ")")
                .toList();
    }
}
THE CLOSED INC-1: EACH LINE OF THE EVIDENCE CHAIN, AND THE PEER THAT SIGNED ITstafftriage-ajudge-opsjudge-changejudge-securityrisk-leadconsoleexec-aTelemetryDiagnosisRemediationProposalRiskReview x3RemediationDecisionOperatorApprovalRemediationReceiptEvery line was read as PEER_ASSERTED: the peer's key signed it and the peer named the agent.The staff peer signs both the telemetry and the proposal, so the chain shows the peer that hosts Assets and Remediation.
Who signed the evidence. The recorder reads every entry about the incident with its issuer and its attestation, and prints the signer on each line. 8 peers appear in INC-1's chain. Giving each agent its own certified key, as example 13 does, would turn the staff peer's two lines into two named agents.

Every line in the recorded run ends in PEER_ASSERTED. The peer's key signed the entry and the peer named the agent. The staff peer hosts four agents, so its two lines, the telemetry and the proposal, both read by staff. Giving each agent a certified key of its own, as example 13 does with PeerIdentity.subordinate, would make those lines AGENT_ATTESTED and name Assets and Remediation separately. Per-agent keys would also let a vote count per agent, which the demo's granularity does not.

What to watch for

The record is a snapshot: the recorder reads its own replica when the receipt arrives, and the record is itself an entry with a 12 hour lease. It reports the incident and does not archive it.

aspace://guide/opscenter/planner · chapter 13

The planner seat: Embabel over the fleet

The twelfth peer has no agents. It reads the fleet's cards. RemoteActions turns each foreign card into an invocable action: invoking the triage action writes an Incident into the incidents space and waits for the Diagnosis that shares its incidentId. The planner holds the incidents, evidence and actions spaces, a route that sends incidents to incidents, and a result route for every type the fleet produces. RemoteActions waits for a result in the space of the input unless a result route says otherwise, so the table in Bridge.RESULT_SPACES decides whether an invocation finds its answer: the assets action takes an Incident from incidents and answers with a Telemetry in evidence, and the dispatcher action takes an OperatorApproval from evidence and answers with a RemediationOrder in actions. The first fleet test invokes the triage action with a fresh incident, INC-3, checks that every generated action awaits its result in the space the table names and carries a description, and invokes the assets action to watch its telemetry arrive in evidence. The second fleet test invokes the dispatcher action and receives its order from the actions space.

Bridge
/**
 * The space each agent writes its result into, by result type, which is where an invocation
 * must wait for it: RemoteActions waits in the input's space unless told otherwise. Every
 * action the fleet's cards generate has its output here. Triage, assets, remediation, the
 * officer, the executors and the recorder answer in evidence; the dispatcher answers in
 * actions, so the planner holds that space too.
 */
public static final Map<Class<?>, String> RESULT_SPACES = Map.of(
        Diagnosis.class, OpsFleet.EVIDENCE,             // Triage
        Telemetry.class, OpsFleet.EVIDENCE,             // Assets
        RemediationProposal.class, OpsFleet.EVIDENCE,   // Remediation
        OperatorApproval.class, OpsFleet.EVIDENCE,      // WatchOfficer
        RemediationOrder.class, OpsFleet.ACTIONS,       // Dispatcher: resultSpace = actions
        RemediationReceipt.class, OpsFleet.EVIDENCE,    // Executor
        IncidentRecord.class, OpsFleet.EVIDENCE);       // Recorder, both of its closes
THE FLEET'S AGENT CARDS (14)RemoteActionsTHE GENERATED @Agent opsCenterFleettriage x2@SpaceTake on incidentstriage_IncidentIncident > Diagnosis (evidence)triage_Incident_2Incident > Diagnosis (evidence)assets@SpaceNotify on incidentsassets_IncidentIncident > Telemetry (evidence)remediation@SpaceNotify on evidenceremediation_DiagnosisDiagnosis > RemediationProposal (evidence)dispatcher@SpaceNotify on evidencedispatcher_OperatorApprovalOperatorApproval > RemediationOrder (actions)watch-officer@SpaceNotify on evidencewatch_officer_RemediationDecisionRemediationDecision > OperatorApproval (evidence)recorder2 x @SpaceNotify on evidencerecorder_RemediationReceiptRemediationReceipt > IncidentRecord (evidence)recorder_RemediationDecisionRemediationDecision > IncidentRecord (evidence)executor x3@OrderedTake on actionsexecutor_RemediationOrderRemediationOrder > RemediationReceipt (evidence)executor_RemediationOrder_2RemediationOrder > RemediationReceipt (evidence)executor_RemediationOrder_3RemediationOrder > RemediationReceipt (evidence)risk-lead@Propose, @OnDecisionnever invocablerisk-judge x3@Ballot on reviewsnever invocable
Cards become typed actions. Fourteen cards arrive by gossip and ten of them yield eleven actions: the recorder offers two, and the three executor cards give the same action three times. A card yields an action when its kind is invocable, its types resolve to local classes, and the planner holds the space the card reads. Proposals, ballots and decision reactions are never invocable, so the risk lead and the judges drop out. Each action awaits its result in the space Bridge.RESULT_SPACES names for its output type, shown in parentheses. That is why the planner holds the actions space: only the dispatcher's order (shaded) appears there. This is the list once the fleet has settled; the demo bridges as soon as the triage card arrives.

EmbabelRemoteActions then generates an Embabel @Agent class whose methods are those actions, one typed @Action per entry, named from the card and the consumed type. It reads Embabel's annotations by name, so the app compiles without Embabel, and under -Pembabel it deploys the generated agent onto the platform beside the local one. The local planner knows two actions, raise (report to incident) and brief (diagnosis to briefing), and cannot connect them. The fleet's triage_Incident closes the gap.

OpsCenterEmbabelApplication
@Agent(description = "Turns an incident report into a briefing with the operations center fleet")
public static class OpsPlannerAgent {

    @Action
    public Incident raise(IncidentReport report) {
        return new Incident(report.incidentId(), report.source(), report.summary(), report.severity());
    }

    @AchievesGoal(description = "A briefing on the incident's most likely cause")
    @Action
    public Briefing brief(Diagnosis diagnosis) {
        return new Briefing(diagnosis.incidentId(), diagnosis.cause(), diagnosis.confidence(),
                diagnosis.diagnosedBy());
    }
}
ONE PLANNING SPACE: THE LOCAL ACTIONS OF OpsPlannerAgent AND THE FLEET'SIncidentReportIncidentDiagnosisBriefingraiselocal @Actiontriage_Incidentfleet, a leased round tripbrieflocal @AchievesGoalthe plans Embabel printed, each formulated after the previous action ran:raise -> triage_Incident -> brief | triage_Incident -> brief | briefINSIDE triage_Incident: 2.0 s IN THE RECORDED RUN, 1.2 s OF IT MODEL TIMEwrite Incidentincidents, 1 h leasea triage peer takes it@SpaceTake, 15 sDiagnosis appearsevidencematched by incidentIdshared-field correlationreturned to the plannera typed Diagnosisthe incident lands in the same space as one the console writes, so the center's other agents see ita worker that dies mid-call delays the planner by one lease: the other peer takes the incident and finishes
GOAP plans across the seam. The planner's two local actions cannot reach a briefing alone, because nothing local turns an incident into a diagnosis. With the generated agent deployed beside them, Embabel plans raise -> triage_Incident -> brief, the plan the code printed, and plans again after each step. The middle step runs on the fleet.

Building and running the profile at 0.5.0

The Embabel sources sit in src/embabel/java and compile only under -Pembabel, which adds embabel-agent-starter 1.5.2 and Spring Boot 4.1.1. On 2026-11-04 mvn -Pembabel compile built against AgentSpaces 0.5.0 with BUILD SUCCESS; javac warns that @EnableAgents is deprecated and marked for removal. The application ran with scripted models and printed the plan below. It excludes the AgentSpaces auto-configurations on purpose, because it boards the fleet in code rather than through the starter, and it waits for the triage card before it bridges, because the generated agent is a snapshot of the actions available at that moment. In the run recorded here the other cards had already arrived, so that snapshot held all 11 actions and the generated agent has a method for each. The planner also logs an ERROR line for each generated action that Embabel cannot reach as a goal (8 lines in this run). Every generated action carries @AchievesGoal, because its card names goals, so Embabel treats each one as a candidate goal and reports the ones whose input nothing in the planning space produces: watch_officer_RemediationDecision and recorder_RemediationDecision, whose RemediationDecision input only the risk panel writes. The plan to the briefing is unaffected, and the block below leaves those lines out.

Recorded Embabel run, 2026-11-04
console: http://localhost:7591/
fleet actions discovered: 11, including triage (Incident -> Diagnosis)
Embabel - Deployed agent opsCenterFleet
	description: The operations center fleet's advertised skills: triage, assets, remediation, dispatch, officer review, execution and recording
Embabel - [elegant_rosalind] formulated plan:
  OpsPlannerAgent.raise -> RemoteFleetAgent$45a155a9.triage_Incident -> OpsPlannerAgent.brief
Embabel - [elegant_rosalind] executed action OpsPlannerAgent.raise in PT0.001S
Embabel - [elegant_rosalind] formulated plan:
  RemoteFleetAgent$45a155a9.triage_Incident -> OpsPlannerAgent.brief
Embabel - [elegant_rosalind] executed action RemoteFleetAgent$45a155a9.triage_Incident in PT2.032S
Embabel - [elegant_rosalind] formulated plan:
  OpsPlannerAgent.brief
Embabel - [elegant_rosalind] executed action OpsPlannerAgent.brief in PT0S
Embabel - [elegant_rosalind] goal OpsPlannerAgent.brief achieved in PT2.068105S
briefing: Briefing[incidentId=INC-7, cause=The auth gateway's token-refresh path is failing because its upstream signing credential expired at midnight, confidence=0.74, diagnosedBy=triage-a]
✓ the plan Embabel printed, raise -> triage_Incident -> brief; timestamps, package names and the world-state dumps are left out
mvn -Pembabel compile exec:java -Dopscenter.main=ai.badmonkey.agentspaces.opscenter.embabel.OpsCenterEmbabelApplication
What to watch for

The incident the planner raises lands in the same space as any other, so the rest of the center sees it. Without --watch the process closes the fleet right after the briefing, so the proposal, vote and execution stages have only begun. Run with --watch to keep the fleet up while the center works the incident, and add OPSCENTER_OPERATOR=console to approve it yourself. Every generated action also carries @AchievesGoal, because the card names goals, which gives the planner more than one goal to choose from once the whole catalogue is bridged.

aspace://guide/opscenter/running · chapter 14

Running, testing, and where to go next

The project is a standalone Maven build. agentspaces-dependencies 0.5.0 from Maven Central manages the core artifacts and embabel-agentspaces, and agentspaces-springai 0.5.0 is pinned in the pom because the BOM does not manage it. No checkout of the core is needed.

./run-demo.sh                     # scripted models, simulated operator, exits when both incidents close
./run-demo.sh --watch             # the same, then keeps the fleet and the console up
OPSCENTER_OPERATOR=console ./run-demo.sh --watch   # a person approves from the console
OPENAI_API_KEY=sk-... ./run-demo.sh                # live models through Spring AI's OpenAI client
mvn test

The console is at http://localhost:7591/ with the operator token opscenter-operator. Peers use ports 7800 to 7811 on loopback, so run one demo at a time. run-demo.sh compiles first, because the project sits outside the core reactor and nothing else builds its classes.

What the test asserts

mvn test runs 7 tests. ScriptedModelsTest has 5, and checks that the scripted models answer through Spring AI's structured-output path, including the judges' dissent and the exposed-key scenario. OpsCenterFleetTest has two, and each runs a whole fleet over loopback TCP on a free block of twelve ports, with scripted models, no network and no keys. The first runs the scenario above, the worker death included, then checks the planner's generated actions: each awaits its result in the space Bridge.RESULT_SPACES names, each carries a description, and the assets action returns its telemetry. The second staffs the console (no simulated officer) and checks the gate: a rejected remediation closes as rejected, the console refuses the approval with the reason and audits it, a stray approval and the planner's dispatcher action write no order, an approved remediation waits for the person and then runs once, and the Incidents panel shows each incident's stage throughout. On 2026-11-04 mvn test took 41.5 s in all, 40.5 s of it the two fleet tests, with all seven tests passing.

OpsCenterFleetTest
// Act one: the incident lands, triage-a takes it, and we kill triage-a.
OpsCenter.raise(ops.console(), "INC-1", "db-primary", "disk usage alarm: data volume at 98 percent", "high");
assertThat(ops.crash().awaitTake(Duration.ofSeconds(30))).as("a worker took the incident").isTrue();
String victim = ops.crash().victim();
String survivor = victim.equals("triage-a") ? "triage-b" : "triage-a";
ops.crash().kill();
ops.peer(victim).close();

Diagnosis diagnosis = OpsCenter.await(() -> OpsCenter.first(evidence, Diagnosis.class, "INC-1"),
        Duration.ofSeconds(120)).orElseThrow();
assertThat(diagnosis.diagnosedBy()).as("the surviving worker finished the job").isEqualTo(survivor);
assertThat(diagnosis.cause()).contains("db-primary");

The limits to keep in mind

  • Delivery is at least once outside the actions space, so a diagnosis can in rare cases be produced twice. It is idempotent.
  • Reads come from the local replica, so a peer sees an entry after replication reaches it.
  • A quorum proves three judges agreed, and nothing more. The simulated officer approves whatever the panel approved; a staffed center sets OPSCENTER_OPERATOR=console and waits for a person. The panel binds the person through the console command and the dispatcher; the actions space itself accepts an order from any peer in this open group.
  • The executors' Raft logs live in memory (MemoryLogStore), so a restarted executor rejoins with an empty log. OrderedTakes.over(..., store) takes a FileLogStore per seat when the log has to survive a restart.
  • The fleet-control space exists for the console's "Broadcast directive" form. No agent attaches a DirectiveGate, so nothing honors a directive.
  • The scripted models are rules. They keep the demo and the test deterministic and keyless; a live model changes the diagnosis text and can change a vote.
  • Evidence lines are PEER_ASSERTED, so they name the hosting peer. Per-agent keys (PeerIdentity.subordinate) would make them AGENT_ATTESTED and name the agent.

Where to go next

THE FIVE LAYERS, AND WHAT OPSCENTER STANDS ONagents and the binder@SpaceTake, @SpaceNotify, @Propose, @Ballot, @OnDecision, @OrderedTake; Spring AI ChatClients; the Embabel bridgeLayer 4 capabilitiesvote, ordered log, aggregate, content, remote actionsopscentervote on reviews, the Raft ordered log on actions, remote actionsLayer 3 spacesreplicated tuple spaces, leases, strategiesopscentersix replicated spaces, take leases, a signed entry per factLayer 2 discoverycards, advertisements, locatorsopscenteragent cards, read by the planner's RemoteActionsLayer 1 peeringmembership, gossip, transportsopscentertcp, SWIM membership, gossip; twelve peersLayer 0 identityEd25519 keys, signed documents, groupsopscenteran Ed25519 key per peer, every entry and ballot signed
The layers under the app. Each layer rests on the one below. A highlighted layer is one the app uses directly; the rest still run beneath it and need no configuration here.