Ask the reasoning service, and handle abstentions

Some decisions turn on evidence that rules cannot read. You send those to the reasoning service as typed questions over the evidence, and it answers each question, or abstains and says what evidence would let it answer. You write the code that acts only on answers you can trust and sends everything else to a person.

What you'll build#

• triage.py: a client that asks three questions about an alert and prints each verdict.
• route.py: a function that turns a response into what your application does next, and never acts on an abstention or a low-confidence answer.

Prerequisites#

• Hosted API access: your reasoning API endpoint in HELIXOR_API_URL and your API key in HELIXOR_API_KEY. The key goes in the X-API-Key header.
• Python with httpx installed (pip install httpx).
• Read Escalating to hosted reasoning for when to escalate at all.

The contract#

POST /v1/decide takes a goal, the questions to answer and the evidence to answer them from. The goal must be one the service has admitted: it names which questions it answers and the floor an answer must clear. Each question has a kind:

Each question comes back with a verdict:

p_correct is the held-out accuracy of the decision head that answered, measured on the reliability bin that holds the answer's probability, with Laplace smoothing. It is not the head's raw score. When that bin has no held-out rows, there is no calibrated probability, and the question abstains with not_certified. With require.min_confidence, an answer whose p_correct is below it also becomes a not_certified abstention.

The response also carries composed, the goal's overall disposition with the rule IDs that produced it, composed from answered questions only, and proof_id, which is empty when no question was answered. Keep proof_id with your record, as you keep receipt_hash for embedded decisions.

The tenant and user of a call come from your API key. Leave scope.tenant out; a value that differs from your key's tenant is refused with 403 tenant_mismatch.

Steps#

1. Write the client

   The alert-triage goal, sec.alert_triage.v1, answers questions by question_id: unauthorized, record_explains and attack_kind. Any error status raises, because an error is never an answer.

   import json
   import os
   
   import httpx
   
   API = httpx.Client(
       base_url=os.environ["HELIXOR_API_URL"],          # your reasoning API endpoint
       headers={"X-API-Key": os.environ["HELIXOR_API_KEY"]},
       timeout=10.0,
   )
   
   
   def boolean(question_id, family_id, prompt):
       return {
           "question_id": question_id,
           "family_id": family_id,
           "kind": "boolean",
           "prompt": prompt,
           "options": [{"option_id": "true"}, {"option_id": "false"}],
       }
   
   
   QUESTIONS = [
       boolean("unauthorized", "sec.unauthorized", "Is this unauthorized activity?"),
       boolean("record_explains", "sec.record_explains", "Does a change record explain it?"),
       {
           "question_id": "attack_kind",
           "family_id": "sec.attack_kind",
           "kind": "choice",
           "prompt": "What kind of activity is this?",
           "options": [{"option_id": o} for o in (
               "host_compromise", "account_takeover", "mail_campaign",
               "exfiltration", "config_change")],
       },
   ]
   
   
   def decide(evidence):
       resp = API.post("/v1/decide", json={
           "goal": "sec.alert_triage.v1",
           "evidence": evidence,
           "questions": QUESTIONS,
           "require": {"proof": True},
       })
       resp.raise_for_status()                          # an error is never an answer
       return resp.json()
   
   
   def show(title, d):
       print(f"== {title}")
       for qid, v in d["verdicts"].items():
           if v["verdict"] == "answer":
               print(f"  {qid:16} answer  {v['value']!r:18} p_correct={v['p_correct']}"
                     f"  proof={v['proof']['kind']} {v['proof']['citations']}")
           else:
               print(f"  {qid:16} {v['verdict']:7} reason={v['reason']}"
                     f"  masked={v['masked']}  would_unmask={v['would_unmask']!r}")
       c = d["composed"]
       print(f"  composed: {c['verdict']} {c['rule_ids']} ({c['band']})")
       print(f"  proof_id: {d['proof_id']!r}  latency_ms: {d['latency_ms']}")
   
   
   if __name__ == "__main__":
       show("alert with a change record", decide([
           {"id": "ALERT-902", "source": "syslog",
            "text": "Database migration running as part of scheduled release ticket CHG-4819"},
           {"id": "CHG-4819", "source": "change-log",
            "text": "Approved change ticket CHG-4819 for database schema migration"},
       ]))
       show("vague alert", decide([
           {"id": "ALERT-977", "source": "edr",
            "text": "Unusual activity on host web-07, please investigate"},
       ]))
   

2. Ask about two alerts

   export HELIXOR_API_URL=...   # your reasoning API endpoint
   export HELIXOR_API_KEY=...   # your API key
   python triage.py
   

   == alert with a change record
     unauthorized     answer  False              p_correct=0.9844  proof=distribution_threshold ['ALERT-902', 'CHG-4819']
     record_explains  answer  True               p_correct=0.963  proof=citation_set ['CHG-4819', 'CHG-4819']
     attack_kind      abstain reason=not_certified  masked=[]  would_unmask=''
     composed: close ['HARD-EXPLAINED', 'DISP-CLOSE'] (record explains (p_correct 0.96 >= 0.75))
     proof_id: 'prf_0bcaa9666e1e4419'  latency_ms: 54
   == vague alert
     unauthorized     answer  True               p_correct=0.9844  proof=distribution_threshold ['ALERT-977']
     record_explains  abstain reason=not_certified  masked=[]  would_unmask=''
     attack_kind      abstain reason=not_certified  masked=[]  would_unmask=''
     composed: queue ['DISP-QUEUE'] (unauthorized (p_correct 0.98 >= 0.75) but attack kind not established at the floor: no containment action (to person))
     proof_id: 'prf_94155cc9c7a84181'  latency_ms: 0
   

   The first alert comes with the change record that explains it. The service answers that the activity is explained, cites the record in the proof, and composes close. It abstains on attack_kind with not_certified: the head has no calibrated probability for its answer, so it does not answer.

   The second alert says only "unusual activity". The service answers that the activity is unauthorized, and abstains on record_explains and attack_kind. Because the kind of activity is not established, composed is queue: a person decides, and no containment action is proposed. latency_ms varies; the first call in a process is slower.

   Here is part of the raw response for the second alert:

   {
     "schema": "helixor.decision.v1",
     "verdicts": {
       "record_explains": {
         "verdict": "abstain",
         "value": null,
         "p_correct": null,
         "proof": null,
         "reason": "not_certified",
         "masked": [],
         "would_unmask": ""
       },
       "attack_kind": {
         "verdict": "abstain",
         "value": null,
         "p_correct": null,
         "proof": null,
         "reason": "not_certified",
         "masked": [],
         "would_unmask": ""
       }
     },
     "composed": {"verdict": "queue", "rule_ids": ["DISP-QUEUE"],
                  "band": "unauthorized (p_correct 0.98 >= 0.75) but attack kind not established at the floor: no containment action (to person)"},
     "proof_id": "prf_8259f834335a4343",
     "cost": null,
     "latency_ms": 52,
     "helix": {"version": "0.1.4", "fingerprint": "", "degraded": [{"component": "encoder", "reason": "encoder_disabled", "detail": "..."}]}
   }
   

   helix.degraded lists components the service ran without for this call. The local instance used here had its encoder disabled; check this list before trusting a response in production.

3. Route on verdicts, not on the composed disposition

   Decide, for each disposition, which answers it depends on. Act only when every one of those is an answer whose p_correct clears your floor. Anything else goes to a person, with the reasons.

   from triage import decide
   
   FLOOR = 0.75          # the lowest p_correct you act on without a person
   
   # Which answers each composed outcome depends on.
   NEEDS = {
       "close": ["record_explains"],
       "act": ["unauthorized", "attack_kind"],
       "queue": [],
   }
   
   
   def route(d):
       """Turn one decision response into what your application does next."""
       composed = d["composed"]["verdict"]
       held = []
       for qid in NEEDS.get(composed, list(d["verdicts"])):
           v = d["verdicts"].get(qid)
           if v is None:
               held.append(f"{qid}: no verdict")
           elif v["verdict"] == "refuse":
               return "reject_request", [f"{qid}: refused ({v['reason']})"]
           elif v["verdict"] == "abstain":
               held.append(f"{qid}: abstained, needs {v['would_unmask'] or v['reason']}")
           elif v["p_correct"] is None or v["p_correct"] < FLOOR:
               held.append(f"{qid}: p_correct {v['p_correct']} below {FLOOR}")
       if held or composed == "queue":
           return "analyst_queue", held
       return composed, [f"proof {d['proof_id']}"]
   
   
   cases = {
       "ALERT-902": [
           {"id": "ALERT-902", "source": "syslog",
            "text": "Database migration running as part of scheduled release ticket CHG-4819"},
           {"id": "CHG-4819", "source": "change-log",
            "text": "Approved change ticket CHG-4819 for database schema migration"},
       ],
       "ALERT-977": [
           {"id": "ALERT-977", "source": "edr",
            "text": "Unusual activity on host web-07, please investigate"},
       ],
   }
   for alert, evidence in cases.items():
       d = decide(evidence)
       outcome, why = route(d)
       print(f"{alert}: composed={d['composed']['verdict']:5} -> {outcome}")
       for line in why:
           print(f"    {line}")
   

   python route.py
   

   ALERT-902: composed=close -> close
       proof prf_281bc3a48c1b422a
   ALERT-977: composed=queue -> analyst_queue
   

   The explained alert is closed with its proof ID, because close depends only on record_explains, which was answered above your floor. The vague alert goes to an analyst because the service composed queue. If it had composed act, route would still hold it unless unauthorized and attack_kind were both answers above your floor. proof_id differs from the previous step because each call generates a new one.

4. Handle errors

   A goal the service has not admitted returns 404 with a typed failure, not an abstention:

   curl -s -X POST "$HELIXOR_API_URL/v1/decide" \
     -H "X-API-Key: $HELIXOR_API_KEY" -H "Content-Type: application/json" \
     -d '{"goal": "returns.refund_check.v1",
          "questions": [{"question_id": "eligible", "family_id": "returns.eligible", "kind": "boolean",
                         "prompt": "Is the refund eligible?",
                         "options": [{"option_id": "true"}, {"option_id": "false"}]}],
          "evidence": [{"id": "r1", "text": "Returned after 20 days"}]}' \
     -w '\nHTTP %{http_code}\n'
   

   {"detail":{"failure_code":"GOAL_NOT_ADMITTED","message":"GOAL_NOT_ADMITTED: goal returns.refund_check.v1 is not admitted"}}
   HTTP 404
   

   Status	Cause
   401	No credentials, or invalid_api_key: the key is invalid, revoked or expired.
   403 tenant_mismatch	scope.tenant differs from the tenant of your API key.
   404 GOAL_NOT_ADMITTED	The goal has no admitted manifest, so it cannot be dispatched.
   422	The request does not match the contract, for example a boolean question whose options are not true, false.
   422 QUESTION_NOT_IN_GOAL	A question_id the goal does not admit. The message lists the admitted ones.
   422 PROOF_WAIVER_UNSUPPORTED	require.proof is false. Every answer carries a proof; omit the field or set it to true.
   500 GOAL_MANIFEST_INVALID	A goal manifest on the service cannot be admitted. A service fault; retrying does not help.
   503 HEAD_UNAVAILABLE	A decision head behind the goal cannot be loaded.

   Treat every error as "no decision" and route to a person; never fall back to a default value.

5. Inspect admitted goals and certify your own decision head

   List every goal admitted on the reasoning service — along with its feature mode, calibration provenance and checkable SubjectProofCertificate digests — via GET /v1/decide/goals:

   curl -s "$HELIXOR_API_URL/v1/decide/goals" -H "X-API-Key: $HELIXOR_API_KEY"
   

   In addition to sec.alert_triage.v1, the service admits two certified Matryoshka (MRL 384d) DecisionHeadStudent goals out of the box:

   • clinc.banking_oos.v1 (banking_intent): trained and temperature-calibrated on the public clinc_oos dataset (95.5% raw accuracy, 95.5% selective accuracy at 100.0% coverage, ECE 0.0229, certificate sproof:9750ced17983a004d4be, artifact sdigest:0465cde4f8404caeceb8). Every answered verdict includes proof.certificate_digest and proof.head_artifact_digest, and runs with live CPU 384d sentence embeddings (helix.degraded: []).
   • ops.domain_selection.v1 (supply_chain_assessment, customer_support_assessment): certified MRL 384d heads achieving 91.5% raw accuracy and 95.9% selective accuracy at 85.5% coverage (ECE 0.0520, certificates sproof:068bc27191eabfcafdba and sproof:cde77306d11d4836dc33) on the supply-chain and customer-support domain selection contract.

   To distill, temperature-calibrate, verify and optionally admit a custom DecisionHeadStudent in under a second on CPU, send labeled training and held-out calibration examples to POST /v1/decide/heads/certify:

   resp = API.post("/v1/decide/heads/certify", json={
       "goal_id": "returns.refund_check.v1",
       "question_id": "eligible",
       "family_id": "returns.eligible",
       "kind": "choice",
       "options": ["approve", "manual_review", "reject"],
       "examples": examples,             # [{"id": "ex-1", "text": "...", "label": "approve", "split": "train"}, ...]
       "feature_mode": "embedding",
       "embedding_dim": 128,
       "target_selective_accuracy": 0.90,
       "min_coverage": 0.50,
       "max_ece": 0.10,
       "max_brier": 0.25,
       "admit_on_pass": True,
   })
   resp.raise_for_status()
   cert = resp.json()
   print(cert["admission"], cert["proof_id"], cert["artifact_digest"])
   

   If the held-out calibration split fails any gate threshold, the service returns proved: false and admission: "rejected" with failed_reasons, and refuses to admit the goal.

Known issues#

How it works#

The service looks the goal up in its registry of admitted goal manifests. A manifest declares the goal's questions, the proof kinds it accepts and its floor. The service then dispatches to the generic pack manifest behind that goal: each question is evaluated by a certified DecisionHeadStudent (using MRL embeddings, character n-grams or hybrid features) that returns a temperature-calibrated distribution over its options, checks its derived abstention threshold and held-out reliability bins, attaches its checkable SubjectProofCertificate digest, and composes a disposition from the answered questions using declarative rules. No text is generated. When a head cannot support an answer, the question is abstained with the evidence that would change that.

The service is stateless: each call carries its own evidence, and nothing from one call is used in the next.

Limits#

• Only admitted goals dispatch. Use POST /v1/decide/heads/certify with admit_on_pass: true to certify and admit a custom decision head on held-out examples.
• Everything you put in evidence leaves your process. Run the embedded decision first and send only what it permits, as shown in Escalating to hosted reasoning.
• A typed escalation path built into the runtime, which links an embedded receipt to a hosted proof automatically, is Planned.

Next steps#