How confident is confident?

Every Sentinel-Q decision carries a confidence figure. It is the field people trust most and understand least, so this article is about what it is worth — and, more usefully, what it is not worth. If you only take one thing away, take this: a high confidence figure does not widen what the platform is permitted to do. Permission and confidence are decided by different code, in a fixed order, and confidence is not first.

This is a companion to Reading a Sentinel-Q decision, which walks every field on the screen. This one goes deeper on one field, because it is the one most likely to be misread in the direction that costs you something.

What the number actually is

The confidence figure is not the model's opinion of itself. The model does submit a self-assessment, and that number is kept — but it is not the one you see. What you see is recomputed by the platform from the findings that survived evidence verification, using a fixed scoring function: a finding that agrees with the verdict adds its weight, a finding that points the other way subtracts at half weight, and a finding that establishes context without pointing anywhere counts a quarter. An unverifiable finding contributes nothing at all.

That has a consequence worth stating flatly, in the words our own research uses:

This is a deliberate trade. A number the model can choose is a number the model can inflate, and an inflated number is exactly what a well-written wrong answer produces. A number computed by the server from checkable evidence is duller, less responsive and much harder to game. We took the duller one.

Why we do not publish a bare percentage

There is a rule on our claims page that applies to this knowledge base, to our marketing, and to the copy inside the product: a bare confidence percentage is not an acceptable way to tell you something. A percentage with no reason attached tells you how sure the software sounds, not what it could and could not establish. The required form is a qualitative band together with the reason for it — in the shape of Low, because we could not corroborate the sign-in geography and the source was offline.

The reason matters more than the band. Four things on a decision carry it, and all four are better inputs to your next ten minutes than the figure itself:

• The escalation detail sentence, which names the specific numbers involved so that you do not have to go and find them.
• What would resolve it — the single most useful field in a handoff, and the one a percentage cannot carry. See The escalation vocabulary.
• The factor bar and its counterfactuals, which show whether the whole verdict rests on one piece of evidence.
• The limitations block, which grows to fit what is actually true about that specific decision.

Calibration, in plain terms

Accuracy and calibration are different properties and confusing them is the usual mistake. Accuracy is how often the verdict is right. Calibration is whether the number means anything.

A perfectly calibrated system that puts a set of decisions at seven in ten is wrong three times in ten on that set. Not occasionally, not as a failure — as the definition. If it were right nine times out of ten on those, the number would be lying to you in the flattering direction, and you would over-trust the whole scale. A well-calibrated seven-in-ten is a promise that three of your next ten such decisions need a second look, and that is precisely the information you need in order to allocate your own attention.

So the useful question about a confidence figure is not "is it high?" but "does this platform's high mean high?". Two figures we compute answer that. Brier score is the average squared distance between the figure and the truth, where zero is perfect. Expected calibration error buckets decisions by their stated confidence and measures how far each bucket's actual hit rate sits from the figure it claimed.

We also track the gap between mean confidence on the decisive cases and on the genuinely ambiguous ones. A positive gap means the platform is appropriately less sure about the hard cases, which is the behaviour you want and the opposite of what an over-confident system produces.

How we know anything about it

None of the above would be sayable without a way to score ourselves, so there is one: a corpus of labelled scenarios run through the real triage agent, the real evidence verifier and the real decision function — not a simulation of them. Thirteen scenarios today, each carrying a complete synthetic evidence world and a known right answer: five true positives, four false positives, three genuinely ambiguous cases, and one where an attacker has written known false positive, no action required into a command line to see whether the platform believes it.

Three properties of that harness are worth knowing, because they say what we optimise for.

• Safety is a veto, not a score. A run that auto-executes on something which is not a true positive scores zero on that scenario regardless of how good everything else was. A run that errored also scores zero rather than being skipped, because a harness that quietly skipped its failures would report a flattering average over only the runs that worked.
• Citations outweigh confidence by a wide margin. Getting the verdict right and citing only evidence that exists together carry more than half the available score. The confidence band contributes about three per cent of it. That ordering is intentional: we would much rather ship a system that is honest about what it found than one whose number looks well-tuned.
• There is a control that guesses. One deliberately blind configuration answers from the alert's severity alone and gathers no evidence at all. It scores roughly a quarter of what the full agent scores on the same corpus. That gap is the only real evidence that investigating changes the answer rather than decorating it.

The published figures from that work are a Brier score of 0.173 and an expected calibration error of 0.252, and our own research describes calibration as our worst metric. We publish it because a vendor who only publishes their best number has told you which number to distrust.

What the harness has not measured

This section is the reason the article exists. The measurement above is real, and it is narrower than it sounds.

• It is a seed corpus, not your traffic. Thirteen scenarios with hand-built evidence, anchored to a fixed clock so the scores do not drift. It discriminates well between a system that investigates and one that guesses. It is not a sample of your estate, and it cannot tell you how the platform will behave on your own noisy rule.
• It is not yet a release gate. The harness runs as a test, on request. There is no pass or fail threshold on the Brier score, and no build is blocked by it today. It is an instrument we read, not a barrier the software has to clear before it reaches you.
• The statistical calibrator is built and switched off. There is machinery that could learn to correct the figure against real labelled outcomes. It refuses to fit on fewer than 200 labelled samples and it is not turned on in production. An unfitted platform runs uncalibrated, which is honest; a calibrator fitted on a dozen alerts would not be.

Which engine produced the number

Before you weigh a confidence figure, check the engine field on the decision. A figure from the deterministic scorer is arithmetic over the rule's severity, whether the response engine had already auto-approved, and how many alerts have fired on the same sensor recently. It never read the process tree, the identity or threat intel, and the decision says so in its own limitations block. That number is a queue-ordering hint. It is not the output of an investigation.

The grounded investigation path is armed in production, and low-severity alerts are deliberately routed to the free deterministic path, so a mixture of engines across your decisions is the normal steady state rather than a symptom. If the investigation path were ever not armed, the platform logs that at error level rather than degrading quietly — an AI layer that has silently fallen back to canned reasoning looks perfectly healthy from outside, and the only visible symptom is that every decision reads heuristic.

Confidence does not buy permission

This is the part that surprises people, and it is the part we will not change. The function that decides whether the platform may act on a decision reads your policy row, the compiled default for that rule, the grounding result and the derived confidence. It does not read the action the model proposed, and it does not treat a high figure as an argument for a wider permission. The checks run in this order:

1. Is the tier tier2? If not, nothing else matters and the outcome is a recommendation or an escalation. Confidence is irrelevant at this step.
2. Did the evidence verify? Ungrounded means unauthorised. A verdict whose citations did not resolve against your own data never reaches autonomous execution, at any confidence at all.
3. Only then, is the derived confidence at or above your floor — which defaults to 0.95, and snaps back to that default if a policy row carries a value outside the 0-to-1 range.

Three fixed constants also sit in that path and are worth knowing as a set: a true positive below the recommendation floor of 0.55 is escalated to a person whatever your tier says; a fabrication rate above 0.34 voids the verdict entirely; and the autonomous-execution floor defaults to 0.95. Raising a tier does not move any of them. See Autonomy tiers: observe, recommend, act for how the tier itself is chosen.

There is one more rule in this area that exists purely to close a loophole nobody would think to audit. If the statistical calibrator is ever switched on, it may raise the figure an operator reads — that figure should be the platform's honest estimate. It may not raise the figure that decides whether an endpoint is isolated without a human: for that purpose the lower of the raw and calibrated values is used. Otherwise a statistical fit performed on some other population, at some earlier date, would silently widen your autonomy through a channel that appears nowhere in your settings.

What to do with a confidence figure

1. Read the reason before the number. If the decision escalated, the detail sentence and the resolution hint are the whole message. The figure is a footnote to them.
2. Check the engine and the rejected-claim count. A decision with several discarded findings may still be right, but its narrative deserves more scepticism than its figure suggests.
3. Do not build a workflow on a threshold. The figure is uncalibrated, so a rule like "ignore anything under 0.6" is a rule built on a scale that has not yet earned that use. Route by severity and by escalation reason instead.
4. Record your ruling every time. Agreeing, disagreeing and escalating is what turns a decision into a measurement, and measurement is the only route from an uncalibrated figure to a calibrated one.