--- before.md +++ after.md @@ -1,5 +1,6 @@ --- date: '2026-03-18T09:00:00-07:00' +lastmod: '2026-09-16T22:20:00-07:00' draft: false title: "Mission Shaping Is Not Enough" description: "Part 2 of the Mission Shaping series. Even a well-shaped Mission cannot by itself make an open-world agent safe." @@ -15,6 +16,8 @@ - IAM - Security --- + +> **Revision — September 16, 2026.** The runtime-alignment section now separates specific behavioral checks from aggregate trust scores and makes their consequences explicit. It also adds requirements for judge calibration, unavailable assessments, bounded retries, and decision records. [Revision history](#revision-history). [Part 1](/notes/the-mission-shaping-problem/) argued that approved intent does not become governable authority on its own. In structured domains, Mission shaping can produce a bounded Mission Authority Model and a reviewable approval object. That is real progress. The Mission is a new authorization primitive: the object that fills the gap between approved intent and governable authority. @@ -255,59 +258,46 @@ If Mission shaping defines the declared authority ceiling, runtime trust defines how much residual uncertainty the system is willing to tolerate inside it. What follows is one practical control-plane pattern, not a settled standard. Continuous observability is not governance, but it is the runtime evidence layer that makes behavioral analysis and dynamic intervention possible. ```text -+----------------------+ -| Mission authority | -| envelope | -+----------------------+ - | - v -+----------------------+ -| Agent actions | -+----------------------+ - | - v -+----------------------+ -| Observability | -| signals | -+----------------------+ - | - v -+----------------------+ -| Trust budget | -+----------------------+ - | - v -+----------------------+ -| Pause / callback / | -| suspend | -+----------------------+ - -+----------------------+ -| Containment | -| boundaries | -+----------------------+ - | - +--> constrain actions - +--> enforce stops +Approved Mission --> Structural authorization ----+ + | | + +--> Required behavioral checks -----------+ + v + Enforced decision before effect + | | + Execute Pause / callback + | + Observed outcomes + | + Trust budget / review + | + Continue / suspend / terminate + +Containment constrains execution and enforces stops throughout. ``` This should be understood as part of the same control plane, not as a second hidden policy engine. The Mission authority artifact remains authoritative for what the Mission permits. The runtime trust layer does not expand authority. It only decides whether the system is still willing to rely on the agent's local judgment within already-shaped bounds. -Even conceptually, that assumes a lot from the underlying system. To score runtime alignment, the control plane needs structured tool calls, stage context, parent Mission linkage, and normalized provenance about where the action came from. Most enterprises do not have that by default, and the visibility gap varies by agent type. Homegrown agents expose different control surfaces than SaaS agent platforms or local workstation agents, and locally run tools are often the hardest for enterprise controls to observe reliably. +Even conceptually, that assumes a lot from the underlying system. To assess runtime alignment, the control plane needs structured tool calls, stage context, parent Mission linkage, and normalized provenance about where the action came from. Most enterprises do not have that by default, and the visibility gap varies by agent type. Homegrown agents expose different control surfaces than SaaS agent platforms or local workstation agents, and locally run tools are often the hardest for enterprise controls to observe reliably. And visibility is not enough. If the agent runtime, orchestration layer, or tool adapter can freely invent the telemetry that the trust layer depends on, the system has recreated self-governance one layer down. Runtime alignment needs integrity as well as visibility. Telemetry must be attributable to a verifiable source, not taken on the agent's word. [SPIFFE/SPIRE](https://spiffe.io/) workload identity provides one foundation: cryptographic attestation of which runtime component produced a given report. [CAEP (Continuous Access Evaluation Protocol)](https://openid.net/wg/sse/) and the Shared Signals Framework provide the transport: a standardized event stream for runtime security events flowing from the components that observe them to the Mission state owner that needs to act on them. Without both integrity and a signal pathway, the runtime alignment layer either trusts what agents report about themselves or polls blindly. -The right model treats residual runtime trust not as binary, but as a continuously scored, consumable resource. The pattern is [just-in-time trust](https://softwareanalyst.substack.com/p/the-future-of-just-in-time-trust) applied to Mission lifecycle. The system grants only the minimum authority needed at the moment it is needed and reconfirms rather than assumes continuity across the task's execution. Recent runtime-governance work, including [MI9](https://arxiv.org/abs/2508.03858), converges on a similar point: semantic telemetry, continuous authorization monitoring, drift detection, and graduated containment become necessary once agent behavior cannot be fully governed at approval time. - -Each agent action is evaluated against a weighted intent score: does this align with the approved intent? Is it consistent with the agent's established pattern for this task? How much damage if it results from context taint? How much of the Mission window remains? A low intent score on a high-risk action is grounds for pausing and requiring re-confirmation even when the action falls within the Mission authority envelope. - -Each Mission carries a consumable trust budget. Actions that score low on intent alignment consume budget faster. Actions anomalous relative to the behavioral baseline consume faster still. Budget exhaustion triggers re-confirmation or Mission suspension. Budget state belongs with the Mission state owner. Exhaustion is a lifecycle event, not a side-channel signal. - -Practical calibration needs a signal model, a baseline, and a cold-start policy. Useful observable signals include resource type distribution relative to the declared purpose, rate of external communication attempts relative to task phase, and semantic similarity of each action to prior approved executions of the same task class. The baseline for a given task class is most reliably established from reviewed template Missions. Prior human-reviewed executions define the expected action distribution that a new run is scored against. The cold-start problem has no clean solution. A reasonable default is to open with the strictest callback thresholds in the relevant template class and relax the budget as an execution baseline accumulates. +One way to aggregate residual runtime uncertainty is to treat continued reliance as a consumable budget. It complements explicit checks at consequential boundaries. The pattern is [just-in-time trust](https://softwareanalyst.substack.com/p/the-future-of-just-in-time-trust) applied to Mission lifecycle. The system grants only the minimum authority needed at the moment it is needed and reconfirms rather than assumes continuity across the task's execution. Recent runtime-governance work, including [MI9](https://arxiv.org/abs/2508.03858), converges on a similar point: semantic telemetry, continuous authorization monitoring, drift detection, and graduated containment become necessary once agent behavior cannot be fully governed at approval time. + +Runtime alignment should begin with explicit questions at consequential checkpoints: does the proposed action serve the approved purpose, is its supporting evidence sufficient, and does its content introduce a risk that structural authorization cannot assess? Keeping these judgments separate makes it possible to explain why execution stopped and what would permit it to resume. + +Policy defines the consequence of each result. Insufficient evidence can request further investigation. A suspected purpose mismatch can pause execution for review. An unavailable or inconclusive assessment must have a defined outcome; where a check is required, silence cannot count as a pass. Retries need limits so uncertainty cannot produce an unbounded loop. When assessments disagree, policy should define whether to gather evidence or escalate, rather than automatically treating a majority as sufficient. + +A trust budget can aggregate residual uncertainty across a Mission, but it must not average away a failed mandatory check. Budget state and lifecycle decisions belong with the Mission state owner. Budget exhaustion triggers re-confirmation or suspension. Favorable judgments never enlarge the approved authority. + +Practical calibration needs a signal model, a baseline, and a cold-start policy. Useful observable signals include resource type distribution relative to the declared purpose, rate of external communication attempts relative to task phase, and semantic similarity of each action to prior approved executions of the same task class. The baseline for a given task class is most reliably established from reviewed template Missions. Prior human-reviewed executions define the expected action distribution that a new run is scored against. The cold-start problem has no clean solution. A reasonable default is to open with the strictest callback thresholds in the relevant template class. Any later relaxation should follow reviewed evidence and a governed policy change, rather than simply accumulating favorable scores. + +Before a judgment controls consequential execution, compare it against independently reviewed examples of the decisions it will govern. Measure harmful passes and unnecessary blocks separately. Record the Mission version, evaluated action, evidence references, judge and rubric versions, verdict, and resulting intervention. Reassess the judge when its model, criteria, or operating conditions change. + +Deterministic orchestration makes the treatment of a verdict explicit. It does not remove uncertainty from the verdict itself. Semantic drift detection is the operational form: monitoring whether the agent's actions are drifting from the semantic neighborhood of the approved intent. A board-packet preparation Mission should produce mostly reads, comparisons, and internal assembly operations. If the action log starts showing HR system reads, treasury queries, or external communication attempts, the distribution has shifted outside the expected neighborhood for this purpose class even if each individual call was permitted. -The governance boundary is precise. Shaped authority is authoritative for permit and deny decisions. Trust scores are authoritative for pause, callback, and suspension decisions. Trust scores do not expand Mission scope. +The governance boundary is precise. The approved Mission bounds the authority a PDP may permit. Runtime assessments can impose additional restrictions, and governing policy translates their results into pause, callback, or suspension decisions. Neither a score nor a budget can override an authorization denial or expand Mission scope. A trust budget is not additional privilege. It is a continued-reliance threshold for exercising already-shaped authority. @@ -385,3 +375,16 @@ For open-world tool-using agents, organizations should expect to lead with containment and treat Mission shaping more as a governance record than as the primary safety guarantee. In some task classes, that should be an explicit policy decision: choose containment-first governance rather than waiting for semantic shaping to mature, because survivability matters more than semantic richness. In other cases, the right policy may be stricter still. Do not allow the task to run agentically at all until the runtime is sufficiently contained and the blast radius is acceptable. The evidence so far also suggests that a shared semantic layer for shaped Mission authority is more likely to emerge as domain-specific families than as one universal model for all agent behavior. That is a more demanding view of what agent authorization requires. It is also the more operationally honest one. + +# Revision History + +**September 16, 2026 — Substantive update.** + +- Replaced the universal weighted-intent-score framing with separate checks at consequential boundaries and made trust budgets an optional aggregation mechanism. +- Defined handling for insufficient evidence, inconclusive or unavailable assessments, disagreement, and bounded retries. A failed mandatory check cannot be averaged away. +- Added independent calibration, separate measurement of harmful passes and unnecessary blocks, and versioned decision evidence. +- Updated the runtime diagram and clarified that governing policy translates assessments into interventions without expanding Mission authority. Relaxation of thresholds requires reviewed evidence and a governed policy change. + +[Version preserved before this revision](/revisions/mission-shaping-is-not-enough/2026-09-16/) · [Exact diff](/revisions/mission-shaping-is-not-enough/2026-09-16/changes.html) · [Revision source](/revisions/mission-shaping-is-not-enough/2026-09-16/after.md) · [Source hashes and provenance](/revisions/mission-shaping-is-not-enough/2026-09-16/manifest.json). + +The preserved version is the repository text immediately before this revision, not a claim about the original publication or its deployment date. Earlier editorial history is not reconstructed here.