Native language lowering benchmark
The domain-trained native model selected the expected assessment on 221/233 (94.85%) frozen synthetic cases. Its calibration gate did not qualify, so no requests were admitted for execution.
Candidate accuracy is not production qualification
The checkpoint remains unbound. The threshold is null, admitted coverage is zero and admitted precision is undefined. The result does not establish general English understanding or real-business error rates.
Task and result#
Given an application-selected order, domain vocabulary and an English request, choose fulfillment, expedite or unsupported. This is selection of a current assessment. It does not measure arbitrary entity extraction, live-source retrieval, policy correctness or action execution. The model has 421,293,827 parameters; 1,685,197,088 bytes of float32 weights.
| Measurement | Recorded result |
|---|---|
| Candidate selections matching the labels | 221/233 (94.85%) |
| Supported assessments selected correctly | 106/112 |
| Unsupported requests rejected | 115/121 |
| Complete contrast pairs correct | 55/64 |
| Admitted cases | 0 |
| Admitted coverage | 0% |
| Precision among admitted cases | Undefined: no admitted cases |
| Warm provider inference, median / p95 | 379.35 / 404.63 ms |
Recorded 2026-10-03 on M5 Max, macOS arm64, Python 3.12.12, 4 CPU threads, batch size 1. Synthetic labels. Provider inference, tokenization and candidate validation; excludes first call, live evidence retrieval, reasoning and business action execution. Evidence and reproduction limits.
Warm latency excludes the first request and uses the evaluator's lower-rank p95 convention. It is not the microsecond rule-evaluation measurement on the Decision Runtime page; the two measurements cover different operations.
Distinguishing features#
| Distinguishing feature | Correct | Cases |
|---|---|---|
| action vs question | 30 | 32 |
| ambiguous | 25 | 25 |
| colloquial | 46 | 48 |
| historical vs current | 31 | 32 |
| negated vs requested | 30 | 32 |
| outside | 32 | 32 |
| single vs multiple | 27 | 32 |
The 12 disagreements comprise supported requests rejected as unsupported and unsupported requests assigned a supported assessment. Quality-hold wording and compound requests are useful next development targets. That hypothesis has not yet been tested. The published observations include every prompt, expected label, selected label, score vector, category and timing for this model.
Calibration result#
The preset requirement is 98% observed precision with at least 30 admitted calibration cases. Temperature fitting and threshold selection use separate calibration partitions. The best subset meeting the size requirement is 31/32 (96.875%) at 0.9838; the stricter error-free subset is 27/27 at 0.9841. Neither outcome qualifies under the full requirement. No test-driven threshold relaxation or binding followed.
Rejecting every request avoids admitted errors on this test but provides no useful admitted coverage. Both facts belong in the result. A confidence score does not verify the interpretation's meaning or the underlying business facts.
Data and selection#
| Partition | Cases |
|---|---|
| Train | 468 |
| Development | 115 |
| Temperature | 178 |
| Gate | 176 |
| Test | 233 |
| Quarantined | 30 |
These are synthetic cases written by a language model and labeled by a second model without seeing its partner's label. Author and judge roles were reversed for the test. Their agreement is same-vendor synthetic consensus, not independent human business ground truth. Related contrast families stay together; business settings are separated across partitions. Overlapping families were quarantined before inference.
Development macro-F1 selected the checkpoint before test evaluation; negative log likelihood was the tie-breaker. Temperature and gate data were not used as the final test. The test is now exposed regression data. Correlated contrast pairs and the deliberately challenging distribution limit extrapolation to production.
Native-path evidence#
Recorded instrumentation observed encoder vectors, finite option vectors and logits. Generation, tokenizer decoding, model-output JSON parsing and inference network entry points were blocked. Application code serialized the resulting typed candidate. These checks support the exercised native path; they do not prove every possible adapter or future configuration has the same behavior.
The subsequent provider abstraction regression reused the same frozen inputs and admission rules. It measures preservation of behavior, not a fresh accuracy improvement.
Evidence and reproduction limits#
Download the evidence manifest and follow Audit a native-model evaluation. The public script checks file digests and recomputes counts, contrast slices, calibration diagnostics and timing aggregates from the published observations. The manifest identifies the source revision, original test digest, language contract and exact weights digest.
The public files are a selected export of the recorded run: provider/vendor identifiers, private paths and service configuration are omitted. Data-file hashes identify this export; the original test and prediction hashes are retained separately. Full inference reproduction requires separately controlled source, base-model/tokenizer resources and the trained checkpoint. This page is a recorded-observation audit, not a publicly reproducible model release or a new run.