A Capable Agent Is Not Eligible for Every Case
A clinical AI agent that is right about ninety percent of the time sounds like something to deploy or not deploy. A study published this week in Nature Medicine treats it as neither. It asks, for each case, whether this agent's diagnosis should be handled autonomously now — and answers with a signal that is observed across the agent's behavior rather than taken from the agent's own confidence. That move, from a property of the agent to a property of the case at decision time, is the shape execution governance takes wherever it appears.
The claim
An agent’s capability is a property of the agent. Whether the agent may act is a property of each case, at the moment the case is in front of it. The two are measured differently, they move differently, and a system that treats the first as if it settled the second has skipped the decision that matters.
A study in Nature Medicine this week makes that distinction operational in one of the most consequential settings for autonomous AI — clinical diagnosis — and, in doing so, shows what an execution boundary has to consult and what it must not rely on.
What the study did
Zhang, Kather and colleagues built a clinical agent that runs entirely inside the institution — no cloud, no external model provider — and conducts a full diagnostic encounter: dialogue with a simulated patient, ordering of tests, retrieval of results, a final diagnosis with a reasoning trace. On the primary benchmark, derived from real intensive-care records, the best on-premise model reached 90.0% accuracy, within a point of a leading cloud model. On a second benchmark, 83.8%.
They did not then ask whether ninety percent is good enough to deploy. They asked a different question: for a given case, can we tell at decision time whether this diagnosis is one the agent should be allowed to handle on its own, or one that should go to a clinician?
To answer it they measured three kinds of reliability signal for each case. The agent’s internal likelihood — how probable the model found its own output. The agent’s expressed certainty — how much it hedged in language. And the agent’s behavioral consistency — whether, run five times independently on the same case, it kept arriving at the same diagnosis. Then they used the best of these signals as a gate. The reliability analyses held the workflow fixed on one model, GLM-4.5-Air, whose repeated-run baseline accuracy on the primary benchmark was 90.6%; the 90.0% figure above is the best single-model result across the on-premise models compared.
What it established
Three findings, in the order that matters for governance.
First, the gate works, and it works by refusing. At a consistency threshold of 0.90, the agent handled 49.4% of cases autonomously — 272 of 551 — with 98.9% accuracy in that subset and three residual errors. The other half went to review, and that stream concentrated the remaining 49 errors. The authors’ own phrasing: selective autonomy “does not remove uncertainty but redistributes it.” In the selective-autonomy analysis, roughly half of the cases were deliberately not converted into autonomous handling, despite the agent’s roughly ninety-percent baseline diagnostic performance. That is the design working, not failing.
Second, the signal that worked was not the agent’s confidence. Behavioral consistency discriminated correct from incorrect diagnoses far better than the model’s internal probability or its linguistic certainty. The stress test is what makes this decisive. When the researchers withheld the patient’s history — removing the evidence the diagnosis rested on — accuracy fell by twenty points. Consistency fell with it. Internal probability and expressed certainty shifted far less, and for one condition both rose while accuracy dropped. They kept some ability to rank correct against incorrect decisions; what they did not do was fall in absolute terms when the evidence was removed. Read at face value, the agent’s confidence signals remained comparatively high even as the evidential grounding of its decisions deteriorated. Only the observed behavior tracked the loss in the way a gate would need.
Third, the threshold is not a constant. Changing the sampling temperature, the number of repeated runs, or the encoder used to compare outputs shifted where 0.90 sat on the accuracy–coverage curve. The authors are explicit that “absolute consistency thresholds are not universal and must be calibrated to the deployment configuration.” Eligibility, on this evidence, is a function of the conditions under which the agent is operating — not a number that travels with the model.
Why this is the execution boundary
Read as AI research, this is a paper about uncertainty estimation. Read through the boundary, it is a demonstration of three properties that any execution decision has to have, in a domain where the cost of getting them wrong is a patient.
Eligibility is decided per case, at decision time. The study never asks “is this agent trustworthy.” It asks, case by case, at the moment the diagnosis is ready to be acted on, “is this one eligible for autonomous handling.” A benchmark score is history — a statement about how the agent performed on cases it has already seen. The decision in front of the system is about the case it has not. This series has made the same argument about approvals, mandates, and certifications: what was established before is an input to the present decision, not a substitute for it. Here the input is a capability score, and the substitution is exactly as tempting.
The signal must be observed, not taken from the agent’s own confidence. The three signals differ in where they come from. Internal probability is generated from inside the same inference process that produced the diagnosis. Linguistic certainty is expressed by the agent in its own words. Behavioral consistency is observed across repeated, independent executions. Only the last is a property of the agent’s behavior rather than of a single run’s self-assessment. The stress test showed why the distinction is not academic: when the evidence was removed, the agent’s confidence signals did not reliably move with the loss of grounding, while behavioral consistency did. An execution boundary should therefore not treat the acting system’s own confidence as sufficient evidence that grounding still holds. It needs a signal that moves when the world moves — a property of the current situation, not only a reading from inside the agent. This is the same reason a scope match cannot stand in for a state check, and a prior assessment cannot stand in for a current finding: an assertion made from a prior or interior position does not carry information about the present.
The threshold belongs to the configuration, not the model. Because the same 0.90 meant different things under different temperatures and encoders, the study’s authors refuse to publish it as a portable number. Translated: the condition under which an action is eligible depends on the operating environment the decision is made in. Move the agent to a different configuration and the eligibility condition has to be re-established there. This is the execution boundary’s fourth question — is the environment the action will execute into still the one the rule was set for — appearing in a paper that never uses the phrase.
The residual tail matters too. Even at strict thresholds, autonomous-stream errors did not reach zero, and the authors say plainly that confidence-based routing “is a mechanism for risk stratification, not a substitute for continuing clinical governance.” The gate reduces exposure; it does not end the obligation to keep deciding.
What this does not show
The study is a retrospective simulation on benchmark cases, not a deployment; no patient was diagnosed by the agent. The gate does not determine whether the agent may generate a diagnosis; it routes an already produced diagnostic output either to autonomous handling or to clinician review. Nor does it authorize treatment — the agent’s output enters a simulated workflow, not a patient’s body. And the authors flag an age gradient in accuracy that a fixed threshold could turn into a fairness problem, which they leave as an open audit question. Behavioral consistency is also not a free external oracle: in the principal analysis it required five independent stochastic runs per case, and the authors report roughly five-fold agent token use as a result. The study establishes that such a signal can support a boundary — not that this particular signal is the only, or a universally practical, way to build one. None of this is an execution-boundary case in the sense this series uses for real events. It is a controlled demonstration of the structure such a boundary needs, published with the code.
Beyond the clinic
Strip the domain and the three properties survive intact. An agent that is right ninety percent of the time on a benchmark should not thereby be routed to autonomous handling on every case; autonomous handling should be reserved for cases where a signal available at decision time, independent of its own self-report, and calibrated to the environment it is running in, establishes eligibility now. The half it does not handle autonomously is not lost capability. It is the boundary doing what boundaries do — sending what cannot be established to where it can be.
Capability is what the agent brings. Eligibility is what the case, at the moment of decision, permits.
Source: Zhang, L., Wölflein, G., Ferber, D. et al. On-premise medical AI agents for reliable clinical decision-making. Nature Medicine (2026), published 15 September 2026, open access; code at github.com/KatherLab/onprem-medical-agents. Figures quoted are from the paper; the reading is this series’.