Decisions across turns
Many decisions depend on something said several turns earlier: don't ask for income again, it was sent; don't submit, the applicant was just laid off. This benchmark measures how often each approach makes the right move when the right move depends on earlier turns.
What is measured#
Each case is a mortgage-intake session of up to 16 turns. The applicant gives the five facts an underwriter needs in any order, sometimes takes one back (a layoff, a deal that fell through) and gives it again later, sometimes says "that's everything" before the file is complete, and asks unrelated questions in between. After every turn the assistant chooses one move under a written intake policy: ask for one missing fact, submit the file for review, decline to quote a rate, or redirect sensitive data.
A history turn is one whose right move cannot be read from that turn's message alone. The right move for every turn comes from the session's scripted events through the reference scorer, never from a system under test.
Approaches#
Every approach has two jobs. Lowering turns a message into the policy's facts ("the seller backed out" means the property is no longer identified). Decisions choose the move from those facts and the policy. Each row names who does which.
| Approach | Lowering | Memory across turns | Decisions |
|---|---|---|---|
| LLM lowering · LLM decisions, no memory | LLM | None: sees the latest message only | LLM |
| LLM lowering · LLM decisions, conversation resent | LLM | The caller resends the whole conversation every turn | LLM |
| LLM lowering · Helixor decisions | LLM, one message at a time, with no history and no policy | Helixor session | Helixor |
| Helixor lowering · Helixor decisions | Helixor, through phrase lists declared in its intake pack | Helixor session | Helixor |
Results#
| Approach | History turns right | Sessions fully right | Sessions with a take-back fully right | Prompt tokens, first turn | Prompt tokens, last turn | Per turn, p50 |
|---|---|---|---|---|---|---|
| LLM lowering · LLM decisions, no memory | 36.8% | 0.0% | 0.0% | 624 | 619 | 1,691 ms |
| LLM lowering · LLM decisions, conversation resent | 86.1% | 53.3% | 34.5% | 635 | 922 | 1,848 ms |
| LLM lowering · Helixor decisions | 89.5% | 61.7% | 58.6% | 741 | 736 | 1,638 ms |
| Helixor lowering · Helixor decisions | 36.8% | 0.0% | 0.0% | 0 | 0 | 0.27 ms |
Measured 2026-10-02 on 60 sessions, 560 turns. LLM rows use a frontier LLM at low reasoning effort with a strict JSON schema. Latency is per turn from one workstation, including network for LLM calls. See the cases and scorer.
How to read the results#
- LLM lowering · LLM decisions, no memory against LLM lowering · Helixor decisions. The same model lowers the same messages one at a time in both rows. The only difference is whether a Helixor session carries what earlier turns established, which facts were taken back and whether the confirmation still counts, and makes the decision.
- Resending the conversation is the strongest LLM-only setup. Its prompt grows every turn; with Helixor deciding, the prompt stays the size of one message.
- Helixor lowering · Helixor decisions measures Helixor's phrase-list lowering. Most natural wordings miss the declared phrases, so this row is limited by lowering, not by its decisions: on the turns where its facts matched the script, its moves were right on most of them. For natural language, pair Helixor's decisions with an LLM for lowering, as in the third row.
Limits#
- One domain, 60 sessions, one LLM at low reasoning effort. The sessions and the applicant's wordings were written by Helixor; about 40% of messages contain one of the intake pack's declared phrases.
- The applicant is scripted: messages do not react to the assistant's questions. The interactive version is the decision strategy benchmark.
- Helixor's moves come from an intake pack compiled from the written policy, so its rule-following is by construction; what it is measured on is carrying the right state from turn to turn.
Cases and scorer#
| File | Contents |
|---|---|
carryover_policy.md | The written intake policy every approach was given. |
carryover_scenarios.jsonl | The 60 sessions: every message, the facts it establishes or takes back, and its kind. |
reference.py | The reference scorer (standard library only). carryover_gold gives the right move after every turn. |
To score your own system, replay each session's messages in order, record the move after every turn, and compare it with carryover_gold. The Helixor rows need the Helixor runtime; see your account.