Benchmarks

Measured results for Helixor components, with their inputs, conditions and evidence. Each page states whether you can rerun the component or audit published observations. Candidate accuracy, admission and runtime evaluation are different measurements.

Native model · Recorded observations

Language lowering and admission

221/233 (94.85%) candidate selections on a frozen synthetic contract. The calibration gate did not qualify; zero requests were admitted.

Integration regression

Replaceable native providers

466 choices and probability vectors preserved through the shared provider registry, with unchanged admission gates.

Results

Decision Runtime

About 25 µs per in-process decision at the median, streaming per push, cold start and HTTP round trip.

Results

Decisions across turns

The right move when it depends on earlier turns: what a stateless model, a resent conversation and a Helixor session each get right.

Results

Decision strategy

Which question to ask next and when to decide, against the exact optimal policy, with and without question costs.

Method

Methodology

What each metric covers, how runs are set up, and the reporting rules.

Method

Reproduce the results

Download the scripts and measure on your own hardware.

Recorded 2026-10-03 on M5 Max, macOS arm64, Python 3.12.12, 4 CPU threads, batch size 1. Synthetic labels. Provider inference, tokenization and candidate validation; excludes first call, live evidence retrieval, reasoning and business action execution. Evidence and reproduction limits.

Headline results#

Component	Result	Conditions
Decision Runtime, in-process	24.3 µs p50, 35.2 µs p99 per `evaluate()` call; 40,775 to 41,563 calls/s	Runtime 0.3.1 (Python build), built-in pack, Apple M5 Max, Python 3.11.4, single process, load average 2.7, run 2026-10-02
Decision Runtime, sidecar	427 µs p50 per HTTP round trip on loopback	Same machine, one keep-alive connection
Certified `DecisionHeadStudent` (Public CLINC150 Banking + OOS)	95.5% accuracy (MRL-384d and Hybrid MRL-384d + n-gram); 95.5% selective accuracy at 100.0% admitted coverage; ECE 0.0229, Brier 0.0327	Public `clinc_oos` dataset (`plus` config, banking + out-of-scope test split, `N=200`), compact open 384d MRL embeddings, temperature-scaled proper-scoring-rule student, checkable `SubjectProofCertificate` `sproof:9750ced17983a004d4be`, 0.128 ms p50 head evaluation
Certified `DecisionHeadStudent` (`ops.domain_selection.v1`)	91.5% accuracy (MRL-384d) and 93.0% accuracy (hashed n-grams 512d); 95.9% selective accuracy at 85.5% admitted coverage; ECE 0.0520, Brier 0.1393	Supply-chain and customer-support assessment contract with out-of-domain `clinc_oos` negatives (`N=200` held-out test), checkable certificates `sproof:068bc27191eabfcafdba` and `sproof:cde77306d11d4836dc33`, 0.076 ms p50 head evaluation
Frozen unfamiliar-rubric generalization (`frozen_unfamiliar_rubric_v1`)	Decomposed atomic fact extraction + `RuleIR`: 83.3% (`180/216`) zero-shot policy + facts, 100.0% (`216/216`, `32/32` paired policy flips, `0/108` option-order flips) with compiled `RuleIR` + frozen 149M NLI fact extractor; single-pass 421M RLCD encoder gets 47.2% (`102/216`), `0/32` policy flips, and `42/108` option-order flips	Unseen supplier-review and support-triage policies (`N=216` observations, `32` paired conjunction/disjunction/exception flips, 0 training steps) and public 12-paragraph procurement policy (`N=120` cases: `120/120` at 0.184 ms with compiled `PlaybookPack`; `19/120` at 313.97 ms with zero-shot 421M RLCD runtime)

Measured: Built-in data-protection pack, runtime 0.3.1, Apple M5 Max, single process, 2026-10-02; call time, evaluate() end to end, median of three runs. Your hardware and payloads change the result. See the reproducible benchmark.

Coming next#

Area	What will be published	Status
Decision Runtime on server hardware	Linux x86_64 and aarch64, multi-process scaling, memory per engine, payload-size sensitivity	Planned
Native Decision Runtime	The same suite against the compiled runtime	Planned
Rostering solver	Time to a feasible roster on public rostering competition instances, with the dataset and scripts	Planned
Hosted reasoning	Answer, abstain and refuse rates, and calibration on held-out sets	Planned

Runnable benchmarks publish their scripts and environment. Recorded-observation audits publish the observations and scorer, and identify any model resources not distributed with them. See Methodology.