Operational excellence

Run packs the way you run code: version them, test them in CI, promote them through environments, and watch what they decide. This page gives you the log schema, the metrics, the promotion flow and the runbooks.

Log receipts, not payloads#

Write one structured log line per decision. Include the fields you need to explain it later and nothing that repeats the sensitive input:

{
  "ts": "2026-09-27T14:03:11Z",
  "request_id": "req_8f2c",
  "pack_id": "custom.claims_guard.v1",
  "pack_version": "1.4.0",
  "pack_digest": "sha256:9d1e…",
  "action": "redact_and_permit_contact_pii",
  "invariants_passed": false,
  "rule_ids": ["RULE-GDPR-EMAIL-REDACT"],
  "redactions": 1,
  "receipt_hash": "hx_proof_ddaca3a134e0b6e6599f3ecd",
  "latency_us": 19.7,
  "caller_latency_ms": 0.4
}

Never log the input text, triggers[].matched_items or a full result object. See Security.
Add the pack version and digest yourself. The receipt covers the pack ID but not its version. Take the version from your deployment metadata and the digest from the .hxpack file you deployed.
Log errors by type. Log the exception class and your request ID, not the message payload.

Metrics to emit#

The runtime exports no metrics endpoint. Emit these from your application or service wrapper:

Metric	Type	Labels	Why
`helixor_decisions_total`	counter	`pack_id`, `action`	Volume, plus the block/redact/permit mix. A sudden shift usually means a pack or input change.
`helixor_rule_fired_total`	counter	`pack_id`, `rule_id`	Which rules do the work, and which never fire.
`helixor_decision_errors_total`	counter	`error_type`	Every error is a fail-closed block. Alert on any increase.
`helixor_decision_latency_seconds`	histogram	`placement`	Measured at the caller, not from `latency_us`.
`helixor_stream_halted_total`	counter	`pack_id`	Streams stopped by a fatal rule.
`helixor_license_days_remaining`	gauge	`license_id`	From the daily expiry check in Reliability.
`helixor_pack_info`	gauge (1)	`pack_id`, `version`, `digest`	Which pack each instance is running, for mixed-version detection during rollouts.

Distributed tracing and a built-in metrics endpoint are Planned. Until then, wrap the evaluate call in a span of your own. Put only the action, pack ID and receipt hash on the span.

Version and promote packs#

The reviewed playbook is the source of truth. Every environment runs a pack built from a tested commit.

Bump version in the playbook for every change. The runtime carries the version but does not enforce ordering, so your pipeline must.
Identify builds by digest. Compiling the same playbook twice produces different bytes, because each compile uses a fresh random nonce. Record the source commit and the SHA-256 of the .hxpack you promote. Promote that file; do not rebuild it later.
Mind the license binding. A pack opens only with the license it was compiled for. If staging and production use different licenses, compile the same reviewed commit once per license in CI, and test both builds.
Keep an approval record. Record which version and digest are approved for each environment, and require a review to change it.

Test packs in CI with golden sets#

A golden set is a file of inputs with the action you expect for each one. Include at least one input per rule that should fire, one per rule that must not fire, and your known tricky cases: values split across chunks, near-miss numbers that fail the checksum, and long inputs.

#!/usr/bin/env bash
set -euo pipefail
helixor-pack compile --playbook playbooks/claims_guard.yaml \
  --license "$HELIXOR_LICENSE_FILE" --out build/claims_guard.hxpack
sha256sum build/claims_guard.hxpack > build/claims_guard.hxpack.sha256

fail=0
while IFS=$'\t' read -r expected text; do
  actual="$(helixor-pack run --pack build/claims_guard.hxpack \
            --license "$HELIXOR_LICENSE_FILE" --text "$text" | python3 -c 'import json,sys; print(json.load(sys.stdin)["action"])')"
  if [ "$actual" != "$expected" ]; then
    echo "FAIL: expected $expected, got $actual"; fail=1
  fi
done < tests/golden/claims_guard.tsv
exit $fail

Keep golden inputs synthetic. Use the standard test values (123-45-6789, 4111-1111-1111-1111, example.com addresses, 555 numbers), never production data.
Fail the build on any mismatch, and on any rule that no golden input exercises.
Provide the CI license from your secret store at job time. Do not commit it.

The testing policies tutorial builds a full test suite.

Runbooks#

Service will not start#

Read the error type in the container log.
PackUnsealError after a license change: the packs were not rebuilt for the new license. Deploy the pack set built for it, or roll back to the previous pair.
LicenseExpiredError: the grace period has ended. Deploy the renewed license with packs rebuilt for it.
LicenseSignatureInvalidError: the license file was altered or truncated in transit. Restore it from your secret manager.
LicenseEntitlementError: the pack or a solver it uses is not in your license. Check with helixor-pack inspect-license.

License expiring#

Request the renewal as soon as the 30-day warning fires.
Follow the rotation procedure in Reliability.
Confirm helixor_license_days_remaining reflects the new license after rollout.

Instance running the wrong pack#

Compare helixor_pack_info across instances.
If any instance reports the Community pack when you expected your own, stop it. A missing file made it fall back.
Check the mounts, then redeploy.

Sudden change in block or redaction rate#

Check whether a pack rollout coincides with the change (helixor_pack_info).
Check helixor_rule_fired_total to find the rule responsible.
If the pack changed, roll back and add the triggering inputs to the golden set. If the pack did not change, investigate the upstream data source.

Decision errors rising#

Every error is a block, so users are affected. Check the error type.
DeveloperQuotaExceededError in production means a Developer license was deployed. Replace it with your production license.