Answer only when the decision is reliable
A decision service that always answers is wrong some of the time, and you cannot tell which times. A service that can say "not sure" lets you automate the cases it is reliable on and hand the rest to a person. This page shows how to act on that, and how to choose the trade.
Three verdicts, three branches#
Each question you send to the hosted reasoning service comes back as one of three verdicts:
| Verdict | What it means | What your code does |
|---|---|---|
answer | The service chose an option, with p_correct, its estimate that the answer is right, at or above the goal's floor. | Act on it. |
abstain | No option cleared the floor, or the head is not certified for this goal. The reason says which. | Queue the case for a person. |
refuse | The request is malformed or not allowed; the reason says which. | Reject the request; do not retry it unchanged. |
The full request and response, and code that branches on all three, are in Ask the reasoning service, and handle abstentions.
Answering less often to be right more often#
Every decision head trades two numbers against each other. Coverage is how often it answers instead of abstaining; accuracy when it answers is how often those answers are right. Raising the abstain threshold lowers coverage and raises accuracy when it answers, because the cases a head abstains on are the ones it is less often right on. Certified MRL decision heads shows both numbers for each head on the public CLINC150 banking requests.
The calibration error reported beside them says how far the head's probabilities are from the rate at which it is actually right; the lower it is, the more you can treat p_correct as a probability when you set a threshold.
Choosing your own threshold#
The goal's floor decides when the service answers at all; it is part of the admitted goal. Your code can be stricter: automate only answers whose p_correct clears a threshold you choose, and queue the rest with the abstentions.
- Price the two mistakes.
What does a wrong automated answer cost, and what does a case sent to a person cost? A refund approval and a marketing tag need very different thresholds.
- Measure on held-out cases.
Never set a threshold on the examples a head trained on. Coverage and accuracy when it answers are only meaningful on cases it has not seen.
- Watch the queue.
A floor sends a share of traffic to people. Size the team for it before you raise the threshold.
- Know what the probability was calibrated on.
A probability is only as good as the cases it was calibrated against. Probabilistic decisions says, for each Helixor number, whether it is a calibrated probability and on what.
When you do not need a probability#
If the decision is a rule over fields you already have, a limit, an eligibility test or an approval threshold, write it as a rule. It decides the same way every time, names the rule that fired and needs no floor. See Rules.