Decision Runtime results

Measured latency and throughput of runtime 0.3.1 on one machine, three runs per benchmark. Every figure links to the script that produced it and to the raw output of all runs.

Run 2026-10-02Runtime 0.3.1Python 3.11.4

Read these before the numbers

Python build. These results measure the current pure-Python runtime. A compiled native runtime is Planned; its results will be published separately when it ships.
Not an idle machine. The one-minute load average was 2.7 during the runs, so tail values (p99, max) are noisier than they would be on a quiet host. It was 3.1 to 3.4 during the 0.3.0 runs.
One laptop. Use these for order of magnitude and shape. Measure on your own hardware with the published scripts.

Environment#

CPU	Apple M5 Max, 18 cores (6 performance, 12 efficiency)
Memory	48 GB
OS	macOS 26.6.2
Software	Python 3.11.4, runtime 0.3.1 (the released wheel, with `helixor-constraint-network` 1.0.1), pydantic 2.13.5; HTTP: fastapi 0.142.2, starlette 1.7.0, uvicorn 0.54.0, the versions pip installed into a fresh virtual environment
Workload	Five fixed payloads cycled in order: one clean, one SSN + IP, one card + email, one health record ID, one phone + email. Built-in Community pack, no license file. Single process.

Summary#

Medians of three runs. Metric definitions are on Methodology.

Measurement	p50	p90	p99	Throughput
Engine time (`latency_us`)	18.4 µs	20.8 µs	27.8 µs	40,996 evaluations/s (loop)
Call time, `evaluate()` end to end	24.3 µs	26.9 µs	35.2 µs	41,524 calls/s (1 / mean)
Streaming, one `push()` of 8 characters	56.3 µs	68.3 µs	131.9 µs	2 KB stream in 14.4 ms
HTTP round trip, `POST /v1/evaluate` on loopback	427 µs	467 µs	511 µs	2,320 requests/s (one connection, sequential)

Cold start#

Stage	Fresh process (5 runs)
Import `helixor_runtime`	86.4 to 99.6 ms
Construct `HelixorEngine()`	0.37 to 0.51 ms
First `evaluate()`	101.0 to 118.3 µs
Second `evaluate()`	45.5 to 58.0 µs
Import, construct and first `evaluate()`	86.9 to 100.2 ms
Whole process (start, import, construct, two evaluations)	0.12 to 0.13 s wall time

Known issue in 0.3.1: cold start at the target, not reliably under it

The target is a cold start under 100 ms. Import, construction and the first evaluation took 86.9 to 100.2 ms, over the target in one of five runs. In a second set of eight runs at a higher load average (4.1 to 4.5) they took 98.2 to 100.4 ms, over the target in three. Import at initialization, not per request.

What changed from 0.3.0#

The 0.3.0 wheel was installed beside 0.3.1 in a second fresh Python 3.11.4 environment with the same dependency versions, and the scripts were run alternating between the two. The raw results have every run.

Import is about 30 ms shorter. Side by side, import took 97.7 to 99.9 ms on 0.3.1 and 127.6 to 138.1 ms on 0.3.0. python -X importtime shows why: 0.3.0 imported the sales engine on import helixor_runtime, about 31 to 34 ms of the import. 0.3.1 takes the sales engine out of the runtime, so it is no longer imported, and the license code no longer loads the cryptography library's key-serialization modules. The import loads 281 modules, against 310.
The new license search costs nothing measurable. 0.3.1 looks for the license in the explicit path, $HELIXOR_LICENSE_FILE, ./helixor.hxlic and ~/.helixor/helixor.hxlic when you construct the engine. Side by side, construction took 0.41 to 0.45 ms on 0.3.1 and 0.41 to 0.46 ms on 0.3.0.
The HTTP round trip did not change with the release. It is 427 µs p50 here against 457 µs in the 0.3.0 run, but the 0.3.0 wheel measured 413 to 424 µs side by side, and 0.3.1 414 to 416 µs. The difference from the 0.3.0 run is the state of the host. The fastapi patch release (0.142.2 here, 0.142.1 in the 0.3.0 run) made no difference either.
Engine time, call time and streaming are unchanged within run-to-run noise; side by side the two wheels cannot be told apart.

What the numbers mean#

About 25 µs per in-process decision at the median, including license check, receipt hashing and building the result. Budget from this figure, not from latency_us, which is about 6 µs lower because it covers evaluation only.
A loopback HTTP hop costs about 18 times the decision. The sidecar pattern adds roughly 0.4 ms; the evaluation inside it is still about 28 µs. Put the runtime in-process when the caller is Python and latency matters. See Performance.
Each streaming push costs about three evaluations. The session re-checks its look-ahead buffer on every push, and before it releases text it evaluates the parts on each side of the cut, so that no value is split. Pushing larger fragments means fewer checks.
Import time dominates cold start. Import is 86 to 100 ms of a cold start; construction and the first evaluation add well under 1 ms. Most of the import is the runtime's own modules and the validation library (measured with python -X importtime).
Warm up once. The first evaluation in a process is about twice as slow as the second, and several times slower than steady state. Run one clean evaluation at startup.

All runs#

Engine time: `bench_engine_time.py`#

Run	Evaluations/s	Mean	p50	p90	p99
1	40,854	18.20 µs	18.42 µs	20.83 µs	27.79 µs
2	40,996	18.14 µs	18.38 µs	20.75 µs	28.83 µs
3	41,183	18.00 µs	18.25 µs	20.54 µs	27.67 µs

Call time: `bench_evaluate_e2e.py`#

Run	p50	p90	p99	Max	Calls/s
1	24.62 µs	27.25 µs	37.00 µs	285.83 µs	40,775
2	24.25 µs	26.88 µs	35.17 µs	85.46 µs	41,563
3	24.33 µs	26.83 µs	34.62 µs	85.58 µs	41,524

Streaming: `bench_streaming.py`#

Run	push p50	p90	p99	Max	2 KB stream p50
1	56.25 µs	68.33 µs	131.88 µs	273.54 µs	14.40 ms
2	56.67 µs	70.25 µs	134.50 µs	296.50 µs	14.54 ms
3	56.12 µs	67.88 µs	130.75 µs	180.62 µs	14.34 ms

HTTP: `bench_http.py`#

Run	p50	p90	p99	Max	Requests/s	Server engine p50
1	427 µs	470 µs	520 µs	611 µs	2,314	29.0 µs
2	423 µs	459 µs	504 µs	580 µs	2,350	28.1 µs
3	427 µs	467 µs	511 µs	665 µs	2,320	28.0 µs

Cold start: `bench_cold_call.py`#

Run	Import	Construct	First evaluate	Second evaluate
1	99.56 ms	0.51 ms	111.8 µs	58.0 µs
2	86.38 ms	0.38 ms	101.0 µs	47.0 µs
3	86.94 ms	0.38 ms	101.8 µs	49.9 µs
4	88.52 ms	0.37 ms	118.3 µs	53.5 µs
5	87.00 ms	0.38 ms	104.5 µs	45.5 µs

Raw results, including the side-by-side runs with 0.3.0: results-2026-10-02.md.

History#

Earlier results, kept as published:

results-2026-09-29.md: runtime 0.3.0, Python 3.11.4, fastapi 0.142.1, load average 3.1 to 3.4. Medians: engine time 18.8 µs p50, call time 24.8 µs p50 and 35.5 µs p99 (39,413 to 41,921 calls/s), streaming push 58.8 µs p50, HTTP round trip 457 µs p50, import 111.4 to 128.5 ms. With the web stack of the 0.2.1 runs (fastapi 0.123.0, uvicorn 0.40.0) the same wheel measured 406 to 416 µs HTTP p50.
results-2026-09-28.md: runtime 0.2.1, load average 7.5 to 10.5. Medians: engine time 19.6 µs p50, call time 24.9 µs p50 and 48.8 µs p99, streaming push 57.9 µs p50, HTTP round trip 347 µs p50, import 864 to 985 ms.
results-2026-09-28-cold-start.md: cold start only, runtime built from source after the lazy-import fix, before the 0.3.0 release. Import 107.0 to 108.6 ms.
results-2026-09-27.md: runtime 0.2.0, load average 5.6 to 7.1. Medians: call time 22.3 µs p50, streaming push 19.2 µs p50 (before the 0.2.1 streaming fix), HTTP round trip 339 µs p50, import 684 to 728 ms.

Not measured yet#

Multi-process throughput scaling on a server CPU.
Payload-length sensitivity (cost against text size and match count).
Memory footprint per engine.
Concurrent HTTP clients against one service.
Linux on x86_64 and aarch64 server hardware.

These are next; they will be added here with their scripts.