Reliability

Once a pack is loaded, the runtime has no external dependencies, so most failures happen at start-up, where they are easy to catch. This page shows how to catch them, keep the service healthy and change licenses and packs without an outage.

How the runtime fails#

Loading a pack fails closed. If any check fails, the runtime raises an error and evaluates nothing.

The CLI prints the error and exits with status 1, so a container that starts helixor-pack serve exits instead of serving. See the error reference for every error type.

Keep the same property in your code. Treat any exception from evaluate(), a timeout, or an unreachable decision service as a block:

def decide(engine, text):
    try:
        result = engine.evaluate(text)
    except Exception:
        # Fail closed: an error is never a pass.
        return "block", None
    if result.action.startswith("block_") or any(t.severity == "FATAL" for t in result.triggers):
        return "block", result
    return "allow", result      # permit_ or redact_: forward result.remedy.clean_text

Verify what you loaded#

Check at start-up that the engine is running the pack you intended. Stop the process if it is not:

import sys
from helixor_runtime import HelixorEngine

EXPECTED_PACK = "custom.claims_guard.v1"

engine = HelixorEngine.load_pack("/run/helixor/pack/guard.hxpack",
                                 license_file="/run/secrets/helixor/helixor.lic")
if engine.pack_id != EXPECTED_PACK:
    sys.exit(f"refusing to start: loaded {engine.pack_id}, expected {EXPECTED_PACK}")
engine.evaluate("warm-up")   # first call compiles patterns; see Performance

load_pack raises when the pack or license file is missing or the pack does not unseal; it never falls back to the built-in pack. Keep the pack_id check anyway: it catches the wrong artifact mounted at the right path. (In 0.2.0, load_pack could not load sealed packs; that is fixed in 0.2.1.)

What is not checked for you#

• What your rules decide. An invalid pattern or an unsupported rule now fails compilation and loading, but a valid pattern can still match too much or too little. Test every rule against your golden set in CI; see Operational excellence.
• License expiry ahead of time. Once the grace period ends, a running engine with a compiled pack raises LicenseExpiredError on every evaluation, and packs no longer load. Nothing warns you before that except license_status in results. Monitor expiry, as described below.

Health checks#

GET /v1/health needs no token and reports the served pack_id and license tier (engine_edition). It does not report license expiry, and it does not evaluate anything. It is enough for liveness. A dedicated readiness endpoint that reports license state is Planned. (In 0.2.0 it returned HTTP 500 under helixor-pack serve.)

For readiness, probe with a real evaluation of a known, harmless payload. Check the pack ID and the action as well as the status code:

import json, os, sys, urllib.request

EXPECTED_PACK = "custom.claims_guard.v1"
headers = {"Content-Type": "application/json"}
if os.environ.get("HELIXOR_SERVICE_TOKEN"):     # the service requires it when set
    headers["Authorization"] = "Bearer " + os.environ["HELIXOR_SERVICE_TOKEN"]
req = urllib.request.Request(
    "http://127.0.0.1:18734/v1/evaluate",
    data=json.dumps({"text": "readiness probe"}).encode(),
    headers=headers,
)
try:
    with urllib.request.urlopen(req, timeout=2) as resp:
        body = json.load(resp)
except Exception as exc:
    sys.exit(f"probe failed: {exc}")
if body["pack_id"] != EXPECTED_PACK or not body["invariants_passed"]:
    sys.exit("probe failed: wrong pack or unexpected verdict")

Run it as an exec probe. Kubernetes HTTP probes cannot send a POST body. Use a generous start-up probe: loading a pack happens once, before the port opens.

Monitor license expiry#

A license has an end date and a grace period (14 days by default). During the grace period packs still load. After it, they do not, and a running engine with a compiled pack stops evaluating. Check the days remaining every day and alert well before the end date:

#!/usr/bin/env bash
set -euo pipefail
out="$(helixor-pack inspect-license "$HELIXOR_LICENSE_FILE")"     # exits 1 if invalid or expired
echo "$out" | grep -E 'Status:|Valid Until:'
days="$(echo "$out" | sed -n 's/.*ACTIVE (\([0-9]*\) days remaining).*/\1/p')"
if [ -z "$days" ]; then echo "CRITICAL: license in grace period"; exit 2; fi
if [ "$days" -lt 30 ]; then echo "WARNING: license expires in $days days"; exit 1; fi

Schedule this check once a day from your job scheduler, not as a loop inside the service. Emit the day count as a metric; see Operational excellence.

Rotate a license and its packs#

Each pack is sealed to the license it was compiled for. A renewed license is a new license, so every pack must be recompiled or downloaded again for it. Rotate them together as one versioned pair:

1. Receive the new license

   Store it in your secret manager as a new version. Do not overwrite the current one.

2. Rebuild the packs

   On a build agent, run helixor-pack compile for each playbook with the new license, or helixor-pack download-pack for catalog packs. Store the results in your artifact store next to the new license version.

3. Test the pair

   Run the golden test set with helixor-pack run against the new pack and license. See Operational excellence.

4. Roll out

   Deploy the new license and new packs in the same release, with a rolling update. Each new instance must pass the evaluation probe before traffic moves.

5. Retire

   After the rollout, remove the old license version and old packs from every environment.

Roll back a pack#

A rollback redeploys the previous pack and license pair from your artifact store with the same rolling update. It works as long as the previous license is still within its validity window. The runtime does not stop you from loading an older pack version. Keep your own record of which version is approved for each environment, and make rollbacks explicit in change control.

Concurrency#

• One engine per worker process. Evaluation results do not depend on shared state. The engine's decision counters and the Developer-tier rate-limit window are updated without locks, so they are only approximate when threads share one engine.
• Scale with processes, not threads. Evaluation is CPU-bound Python, so threads add no throughput. Run several worker processes, each with its own engine, or several service replicas.
• One streaming session per stream. A session holds the stream's buffer and state. Never share one between streams or threads.
• Decision service replicas. helixor-pack serve runs a single process. Scale it by running more containers.
• Developer licenses are for testing. Their rate limits are counted per process, and exceeding them raises an error. Use an Enterprise license in production.

Dependencies at run time#

After start-up, evaluation needs only CPU and memory. The Helixor compiler service and the hosted reasoning API are not on the decision path. An outage of either one does not affect decisions that are already running. It does affect builds that compile remotely and requests you escalate.