Ask the reasoning service, and handle abstentions

Some decisions turn on evidence that rules cannot read. You send those to the reasoning service as typed questions over the evidence, and it answers each question, or abstains and says what evidence would let it answer. You write the code that acts only on answers you can trust and sends everything else to a person.

What you'll build#

• triage.py: a client that asks three questions about an alert and prints each verdict.
• route.py: a function that turns a response into what your application does next, and never acts on an abstention or a low-confidence answer.

Prerequisites#

• Hosted API access: your reasoning API endpoint in HELIXOR_API_URL and your API key in HELIXOR_API_KEY. The key goes in the X-API-Key header.
• Python with httpx installed (pip install httpx).
• Read Escalating to hosted reasoning for when to escalate at all.

The contract#

POST /v1/decide takes a goal, the questions to answer and the evidence to answer them from. The goal must be one the service has admitted: it names which questions it answers and the floor an answer must clear. Each question has a kind:

Each question comes back with a verdict:

p_correct is the held-out accuracy of the decision head that answered, measured on the reliability bin that holds the answer's probability, with Laplace smoothing. It is not the head's raw score. When that bin has no held-out rows, there is no calibrated probability, and the question abstains with not_certified. With require.min_confidence, an answer whose p_correct is below it also becomes a not_certified abstention.

The response also carries composed, the goal's overall disposition with the rule IDs that produced it, composed from answered questions only, and proof_id, which is empty when no question was answered. Keep proof_id with your record, as you keep receipt_hash for embedded decisions.

The tenant and user of a call come from your API key. Leave scope.tenant out; a value that differs from your key's tenant is refused with 403 tenant_mismatch.

Steps#

1. Write the client

   The alert-triage goal, sec.alert_triage.v1, answers questions by question_id: unauthorized, record_explains and attack_kind. Any error status raises, because an error is never an answer.

   import json
   import os
   
   import httpx
   
   API = httpx.Client(
       base_url=os.environ["HELIXOR_API_URL"],          # your reasoning API endpoint
       headers={"X-API-Key": os.environ["HELIXOR_API_KEY"]},
       timeout=10.0,
   )
   
   
   def boolean(question_id, family_id, prompt):
       return {
           "question_id": question_id,
           "family_id": family_id,
           "kind": "boolean",
           "prompt": prompt,
           "options": [{"option_id": "true"}, {"option_id": "false"}],
       }
   
   
   QUESTIONS = [
       boolean("unauthorized", "sec.unauthorized", "Is this unauthorized activity?"),
       boolean("record_explains", "sec.record_explains", "Does a change record explain it?"),
       {
           "question_id": "attack_kind",
           "family_id": "sec.attack_kind",
           "kind": "choice",
           "prompt": "What kind of activity is this?",
           "options": [{"option_id": o} for o in (
               "host_compromise", "account_takeover", "mail_campaign",
               "exfiltration", "config_change")],
       },
   ]
   
   
   def decide(evidence):
       resp = API.post("/v1/decide", json={
           "goal": "sec.alert_triage.v1",
           "evidence": evidence,
           "questions": QUESTIONS,
           "require": {"proof": True},
       })
       resp.raise_for_status()                          # an error is never an answer
       return resp.json()
   
   
   def show(title, d):
       print(f"== {title}")
       for qid, v in d["verdicts"].items():
           if v["verdict"] == "answer":
               print(f"  {qid:16} answer  {v['value']!r:18} p_correct={v['p_correct']}"
                     f"  proof={v['proof']['kind']} {v['proof']['citations']}")
           else:
               print(f"  {qid:16} {v['verdict']:7} reason={v['reason']}"
                     f"  masked={v['masked']}  would_unmask={v['would_unmask']!r}")
       c = d["composed"]
       print(f"  composed: {c['verdict']} {c['rule_ids']} ({c['band']})")
       print(f"  proof_id: {d['proof_id']!r}  latency_ms: {d['latency_ms']}")
   
   
   if __name__ == "__main__":
       show("alert with a change record", decide([
           {"id": "ALERT-902", "source": "syslog",
            "text": "Database migration running as part of scheduled release ticket CHG-4819"},
           {"id": "CHG-4819", "source": "change-log",
            "text": "Approved change ticket CHG-4819 for database schema migration"},
       ]))
       show("vague alert", decide([
           {"id": "ALERT-977", "source": "edr",
            "text": "Unusual activity on host web-07, please investigate"},
       ]))
   

2. Ask about two alerts

   export HELIXOR_API_URL=...   # your reasoning API endpoint
   export HELIXOR_API_KEY=...   # your API key
   python triage.py
   

   == alert with a change record
     unauthorized     answer  False              p_correct=0.9844  proof=distribution_threshold ['ALERT-902', 'CHG-4819']
     record_explains  answer  True               p_correct=0.963  proof=citation_set ['CHG-4819', 'CHG-4819']
     attack_kind      abstain reason=not_certified  masked=[]  would_unmask=''
     composed: close ['HARD-EXPLAINED', 'DISP-CLOSE'] (record explains (p_correct 0.96 >= 0.75))
     proof_id: 'prf_0bcaa9666e1e4419'  latency_ms: 54
   == vague alert
     unauthorized     answer  True               p_correct=0.9844  proof=distribution_threshold ['ALERT-977']
     record_explains  abstain reason=not_certified  masked=[]  would_unmask=''
     attack_kind      abstain reason=not_certified  masked=[]  would_unmask=''
     composed: queue ['DISP-QUEUE'] (unauthorized (p_correct 0.98 >= 0.75) but attack kind not established at the floor: no containment action (to person))
     proof_id: 'prf_94155cc9c7a84181'  latency_ms: 0
   

   The first alert comes with the change record that explains it. The service answers that the activity is explained, cites the record in the proof, and composes close. It abstains on attack_kind with not_certified: the head has no calibrated probability for its answer, so it does not answer.

   The second alert says only "unusual activity". The service answers that the activity is unauthorized, and abstains on record_explains and attack_kind. Because the kind of activity is not established, composed is queue: a person decides, and no containment action is proposed. latency_ms varies; the first call in a process is slower.

   Here is part of the raw response for the second alert:

   {
     "schema": "helixor.decision.v1",
     "verdicts": {
       "record_explains": {
         "verdict": "abstain",
         "value": null,
         "p_correct": null,
         "proof": null,
         "reason": "not_certified",
         "masked": [],
         "would_unmask": ""
       },
       "attack_kind": {
         "verdict": "abstain",
         "value": null,
         "p_correct": null,
         "proof": null,
         "reason": "not_certified",
         "masked": [],
         "would_unmask": ""
       }
     },
     "composed": {"verdict": "queue", "rule_ids": ["DISP-QUEUE"],
                  "band": "unauthorized (p_correct 0.98 >= 0.75) but attack kind not established at the floor: no containment action (to person)"},
     "proof_id": "prf_8259f834335a4343",
     "cost": null,
     "latency_ms": 52,
     "helix": {"version": "0.1.4", "fingerprint": "", "degraded": [{"component": "encoder", "reason": "encoder_disabled", "detail": "..."}]}
   }
   

   helix.degraded lists components the service ran without for this call. The local instance used here had its encoder disabled; check this list before trusting a response in production.

3. Route on verdicts, not on the composed disposition

   Decide, for each disposition, which answers it depends on. Act only when every one of those is an answer whose p_correct clears your floor. Anything else goes to a person, with the reasons.

   from triage import decide
   
   FLOOR = 0.75          # the lowest p_correct you act on without a person
   
   # Which answers each composed outcome depends on.
   NEEDS = {
       "close": ["record_explains"],
       "act": ["unauthorized", "attack_kind"],
       "queue": [],
   }
   
   
   def route(d):
       """Turn one decision response into what your application does next."""
       composed = d["composed"]["verdict"]
       held = []
       for qid in NEEDS.get(composed, list(d["verdicts"])):
           v = d["verdicts"].get(qid)
           if v is None:
               held.append(f"{qid}: no verdict")
           elif v["verdict"] == "refuse":
               return "reject_request", [f"{qid}: refused ({v['reason']})"]
           elif v["verdict"] == "abstain":
               held.append(f"{qid}: abstained, needs {v['would_unmask'] or v['reason']}")
           elif v["p_correct"] is None or v["p_correct"] < FLOOR:
               held.append(f"{qid}: p_correct {v['p_correct']} below {FLOOR}")
       if held or composed == "queue":
           return "analyst_queue", held
       return composed, [f"proof {d['proof_id']}"]
   
   
   cases = {
       "ALERT-902": [
           {"id": "ALERT-902", "source": "syslog",
            "text": "Database migration running as part of scheduled release ticket CHG-4819"},
           {"id": "CHG-4819", "source": "change-log",
            "text": "Approved change ticket CHG-4819 for database schema migration"},
       ],
       "ALERT-977": [
           {"id": "ALERT-977", "source": "edr",
            "text": "Unusual activity on host web-07, please investigate"},
       ],
   }
   for alert, evidence in cases.items():
       d = decide(evidence)
       outcome, why = route(d)
       print(f"{alert}: composed={d['composed']['verdict']:5} -> {outcome}")
       for line in why:
           print(f"    {line}")
   

   python route.py
   

   ALERT-902: composed=close -> close
       proof prf_281bc3a48c1b422a
   ALERT-977: composed=queue -> analyst_queue
   

   The explained alert is closed with its proof ID, because close depends only on record_explains, which was answered above your floor. The vague alert goes to an analyst because the service composed queue. If it had composed act, route would still hold it unless unauthorized and attack_kind were both answers above your floor. proof_id differs from the previous step because each call generates a new one.

4. Handle errors

   A goal the service has not admitted returns 404 with a typed failure, not an abstention:

   curl -s -X POST "$HELIXOR_API_URL/v1/decide" \
     -H "X-API-Key: $HELIXOR_API_KEY" -H "Content-Type: application/json" \
     -d '{"goal": "returns.refund_check.v1",
          "questions": [{"question_id": "eligible", "family_id": "returns.eligible", "kind": "boolean",
                         "prompt": "Is the refund eligible?",
                         "options": [{"option_id": "true"}, {"option_id": "false"}]}],
          "evidence": [{"id": "r1", "text": "Returned after 20 days"}]}' \
     -w '\nHTTP %{http_code}\n'
   

   {"detail":{"failure_code":"GOAL_NOT_ADMITTED","message":"GOAL_NOT_ADMITTED: goal returns.refund_check.v1 is not admitted"}}
   HTTP 404
   

   Status	Cause
   401	No credentials, or invalid_api_key: the key is invalid, revoked or expired.
   403 tenant_mismatch	scope.tenant differs from the tenant of your API key.
   404 GOAL_NOT_ADMITTED	The goal has no admitted manifest, so it cannot be dispatched.
   422	The request does not match the contract, for example a boolean question whose options are not true, false.
   422 QUESTION_NOT_IN_GOAL	A question_id the goal does not admit. The message lists the admitted ones.
   422 PROOF_WAIVER_UNSUPPORTED	require.proof is false. Every answer carries a proof; omit the field or set it to true.
   500 GOAL_MANIFEST_INVALID	A goal manifest on the service cannot be admitted. A service fault; retrying does not help.
   503 HEAD_UNAVAILABLE	A decision head behind the goal cannot be loaded.

   Treat every error as "no decision" and route to a person; never fall back to a default value.

Known issues#

How it works#

The service looks the goal up in its registry of admitted goal manifests. A manifest declares the goal's questions, the proof kinds it accepts and its floor. The service then dispatches to the engine behind that goal. For alert triage, that is a set of decision heads, one per question, each returning a distribution over its options; the service turns each distribution into an answer or an abstention and composes a disposition from the answers with a small rulebook. No text is generated. When a head cannot support an answer, the question is abstained with the evidence that would change that.

The service is stateless: each call carries its own evidence, and nothing from one call is used in the next.

Limits#

• Only admitted goals dispatch. Admitting your own goal is not yet self-service.
• Everything you put in evidence leaves your process. Run the embedded decision first and send only what it permits, as shown in Escalating to hosted reasoning.
• A typed escalation path built into the runtime, which links an embedded receipt to a hosted proof automatically, is Planned.

Next steps#