Decision strategy
A good decision-maker asks only what can change the outcome, in the order that costs least, and decides as soon as the answers settle it. These benchmarks measure each approach against the exact optimal policy, found by searching every question order.
What is measured#
Each case has hidden yes/no facts and a written policy that maps every combination to one outcome. Each turn the agent either asks about one fact or decides. A simulated counterpart answers whatever is asked, in a self-contained sentence, so the agent's strategy drives the conversation.
- Loan routing. Six applicant facts, a decision list over six loan programs, every question costs the same. In about a third of the cases the applicant changes a deciding fact part-way ("it'll be a rental after all"), so the agent has to re-plan.
- Credit screening, with question costs. Seven verification checks, each with a declared cost and pass rate (a hard credit pull costs ten times a pre-approval lookup). The right order depends on cost against the chance that a check settles the decision.
The reference scorer computes the optimal policy exactly and grades every move: the correct outcome, the cost spent against what the optimal policy spends on the same case, questions whose answer could not change the outcome (irrelevant), questions already answered (repeated), and decisions taken before the answers settled them (premature, even when the guess was right).
Approaches#
Every approach has two jobs. Lowering turns an answer into the policy's facts ("I'm at 612 right now" means credit below 680). Decisions choose the next question or the outcome. Each row names who does which; "exact facts" means the benchmark supplies the facts itself, so lowering cannot be wrong.
| Approach | Lowering | Memory across turns | Decisions |
|---|---|---|---|
| LLM lowering · LLM decisions, no memory | LLM | None: sees the latest message only | LLM |
| LLM lowering · LLM decisions, conversation resent | LLM | The whole conversation, resent each turn | LLM |
| Exact facts · LLM decisions | Exact | Exact | LLM |
| Exact facts · Helixor decisions | Exact | Helixor session | Helixor |
| LLM lowering · Helixor decisions | LLM, one message at a time, with no history and no policy | Helixor session | Helixor |
The two "exact facts" rows get identical, perfect information, so the only difference between them is whose reasoning decides.
Not measured here: Helixor lowering · Helixor decisions
The counterpart's answers come from small, fixed sets of wordings. Any phrase list written for Helixor's own lowering would be fitted to those wordings, so its score would not mean anything. Helixor's own lowering is measured on natural wordings in Decisions across turns.
Loan routing#
| Approach | Right program | Right, no extra questions, not premature | Questions asked | Optimal policy | Repeated per case | Premature | Per turn, p50 |
|---|---|---|---|---|---|---|---|
| LLM lowering · LLM decisions, no memory | 25.0% | 16.7% | 7.88 | 3.13 | 6.25 | 0.0% | 1,921 ms |
| LLM lowering · LLM decisions, conversation resent | 96.7% | 88.3% | 3.33 | 3.15 | 0.20 | 1.7% | 1,803 ms |
| Exact facts · LLM decisions | 100.0% | 93.3% | 3.18 | 3.15 | 0.00 | 0.0% | 1,602 ms |
| Exact facts · Helixor decisions | 100.0% | 93.3% | 3.18 | 3.15 | 0.00 | 0.0% | 0.43 ms |
| LLM lowering · Helixor decisions | 98.3% | 81.7% | 3.43 | 3.15 | 0.27 | 0.0% | 1,139 ms |
Measured 2026-10-02 on 60 routing cases. LLM rows use a frontier LLM at low reasoning effort with a strict JSON schema. Latency is per turn from one workstation, including network for LLM calls. See the cases and scorer.
Credit screening, with question costs#
| Approach | Right decision | Right, at optimal cost, not premature | Check cost spent | Optimal policy | Over optimal | Irrelevant per case | Premature | Per turn, p50 |
|---|---|---|---|---|---|---|---|---|
| LLM lowering · LLM decisions, no memory | 3.8% | 1.2% | 22.18 | 18.62 | +19.1% | 0.01 | 0.0% | 2,193 ms |
| LLM lowering · LLM decisions, conversation resent | 100.0% | 57.5% | 19.52 | 18.62 | +4.8% | 0.00 | 0.0% | 2,124 ms |
| Exact facts · LLM decisions | 100.0% | 58.8% | 19.44 | 18.62 | +4.4% | 0.00 | 0.0% | 2,380 ms |
| Exact facts · Helixor decisions | 100.0% | 100.0% | 18.62 | 18.62 | +0.0% | 0.00 | 0.0% | 1.74 ms |
| LLM lowering · Helixor decisions | 100.0% | 100.0% | 18.62 | 18.62 | +0.0% | 0.00 | 0.0% | 1,521 ms |
Measured 2026-10-02 on 80 screening cases. LLM rows use a frontier LLM at low reasoning effort with a strict JSON schema. Latency is per turn from one workstation, including network for LLM calls. See the cases and scorer.
How Helixor chooses#
Helixor compiles the written policy into a decision pack: one path per combination of answers that settles the outcome. Its routing is therefore correct by construction once the facts are known; the benchmark measures its choice of what to ask next.
- Loan routing uses the default question ranking. Every fact has the same declared weight, so the order comes from the engine counting which still-possible paths an answer could settle. An answer that rules a path out removes that path from the count, and a settled "no" is not asked again.
- Credit screening declares each check's cost and pass rate in the pack and selects expected-cost ranking: the engine searches the pack's own paths for the check that minimises the expected total cost. These are the same costs and rates the LLM rows read in the policy text.
Limits#
- Two small, fully specified policies (60 and 80 cases) and one LLM at low reasoning effort. Larger or less precisely written policies are not covered yet.
- The counterpart is simulated and answers truthfully; the screening cases are drawn from the declared pass rates, the model the optimal policy assumes.
- The optimal policy minimises expected cost over all cases; on an individual case a different order can spend less.
Cases and scorer#
| File | Contents |
|---|---|
routing_policy.md, screening_policy.md | The written policies every approach was given. |
routing_episodes.jsonl, screening_episodes.jsonl | Every case: the hidden facts, what the opening message volunteers, and any scheduled change. |
reference.py | The policies as code, the exact optimal policy and the scorer (standard library only). python reference.py prints the optimal expected costs; python reference.py routing runs the optimal policy through every case. |
To score your own system, play each case with your agent, answering every question from the case's facts (apply a scheduled change right after the given number of questions), and compare its moves and spend with the scorer. The Helixor rows need the Helixor runtime; see your account.