Audit a native-model evaluation

Recompute the reported candidate score and admission outcome from the published observations. This exercise needs only Python and the evidence files; it makes no model or network calls after download.

1. Download the evidence into a new directory#

mkdir native-lowering-evidence
cd native-lowering-evidence
for file in manifest.json predictions.json calibration.json provider-regression.json recompute.py; do
  curl --fail --show-error --location \
    "https://helixor.dev/static/benchmarks/native-lowering/$file" --output "$file"
done
python3 recompute.py

You can also download the manifest, test observations, calibration observations, provider regression and audit script individually. The script checks each data file against its manifest digest before calculating results. These are integrity checks, not cryptographic proof of the labels.

2. Read candidate accuracy and admitted coverage separately#

The recomputed result is 221/233 (94.85%) candidate accuracy. It also reports no accepted cases, a null threshold and null accepted precision. Precision with an empty accepted set is undefined. It must not be reported as perfect precision.

{
  "cases": 233,
  "correct": 221,
  "accuracy_pct": 94.84978540772532,
  "supported_correct": 106,
  "supported": 112,
  "unsupported_rejected": 115,
  "unsupported": 121,
  "pairs_correct": 55,
  "pairs": 64,
  "accepted": 0,
  "accepted_precision": null,
  "coverage_pct": 0.0,
  "threshold": null,
  "best_sized_gate": {
    "threshold": 0.9838,
    "accepted": 32,
    "correct": 31,
    "precision": 0.96875
  },
  "largest_error_free_gate": {
    "threshold": 0.9841,
    "accepted": 27,
    "correct": 27,
    "precision": 1.0
  },
  "median_ms": 379.351083538495,
  "p95_ms": 404.6315000159666,
  "warm_observations": 232,
  "slices": {
    "action_vs_question": {
      "correct": 30,
      "cases": 32
    },
    "ambiguous": {
      "correct": 25,
      "cases": 25
    },
    "colloquial": {
      "correct": 46,
      "cases": 48
    },
    "historical_vs_current": {
      "correct": 31,
      "cases": 32
    },
    "negated_vs_requested": {
      "correct": 30,
      "cases": 32
    },
    "outside": {
      "correct": 32,
      "cases": 32
    },
    "single_vs_multiple": {
      "correct": 27,
      "cases": 32
    }
  }
}

Recorded 2026-10-03 on M5 Max, macOS arm64, Python 3.12.12, 4 CPU threads, batch size 1. Synthetic labels. Provider inference, tokenization and candidate validation; excludes first call, live evidence retrieval, reasoning and business action execution. Evidence and reproduction limits.

3. Inspect the errors#

import json
from pathlib import Path

rows = json.loads(Path("predictions.json").read_text())
for row in rows:
    if row["choice"] != row["expected"]:
        print(row["id"], row["category"])
        print(row["text"])
        print("expected:", row["expected"], "selected:", row["choice"])

Review the 12 disagreements in context. The labels are synthetic author/judge consensus, not human-reviewed operational outcomes. A reviewer can disagree with a label; retain that disagreement rather than silently relabeling a test after seeing predictions.

4. Understand why the gate stayed closed#

The declared calibration requirement is 98% observed precision with at least 30 admitted calibration cases. The audit enumerates thresholds on the gate partition only, excluding predictions of unsupported from potential admissions. It uses the recorded score rounding. The result is 31/32 (96.875%) at 0.9838; the stricter error-free subset is 27/27 at 0.9841. The smaller error-free subset does not satisfy the minimum sample count.

The script diagnoses the recorded gate; it never writes a replacement policy or activates a model. Candidate confidence and admission qualification are different facts.

5. Design your own next evaluation#

1. Freeze the language contract, allowed answers and target error/coverage requirements before training.
2. Keep development selection, temperature fitting, gate selection and final evaluation separate.
3. Use distinguishing contrasts and independent business review for the domain you intend to support.
4. Report candidate errors, wrong admissions, admitted precision and useful coverage together.
5. Once you inspect this test's errors, treat it as regression data. Use a new unseen set for the next accuracy claim.

Read the complete benchmark scope and provider refactor evidence before drawing conclusions about production use.