Taking the Clinical Decision Out of the LLM

AI for Psychology, Psychology for AI.

An LLM asked to do therapy improvises. It's trained to be agreeable, and the decisions that matter most in a clinical context — which techniques are on the table, how deep, is this safe right now — are exactly the ones you don't want an agreeable model making.

We tried doing this with prompts first. Prompts don't hold. A sufficiently emotional message talks the model past its instructions, and the failure is silent: you get a fluent, warm, confident reply that happens to be clinically wrong. There's no exception to catch.

So we took the decision out of the model.

The split

Every turn runs the same fixed pipeline:

crisis pre-screen → SCORE → select bucket → admission gate → allowed moves → practice shortlist → GENERATE

If the crisis pre-screen triggers, the pipeline stops there: the message is not scored on that turn, and no reply is generated. A client's first turn with an Avatar, and a calm complaint about the conversation itself, skip the steps between SCORE and GENERATE.

Three of those steps use a model. The rest is Python.

• Crisis pre-screen — a keyword check in plain code runs first; if it finds nothing, a separate model call classifies the message
• SCORE — the model reads the message and emits a state assessment across five dimensions
• GENERATE — the model picks from what Python allowed for this turn and writes the words

Everything between SCORE and GENERATE is deterministic code. Python decides what is allowed; the model never does. It doesn't see the whole admission table, only this turn's cell: the moves it may use and the moves it must not, and sometimes a short list of micro-practices. By the time it's called to generate, all of that has already been computed. In most cells it picks one or two moves; in two cells Python fixes their order instead. It offers at most one micro-practice, or none.

This is the whole idea, and it's not specific to therapy: if a decision has consequences you can enumerate, don't ask the model to make it. Ask the model to characterize the input, decide in code, then ask the model to render the output.

The admission table

Nine therapeutic schools × four client states. The cell says what's admissible.

Concretely: CBT with a regulated client admits a thought record, evidence-checking, a behavioral experiment. The same CBT, the same Avatar, a client who's decompensating — grounding only. The technique set doesn't narrow because the model judged the moment and felt cautious. It narrows because the state score indexed a different row.

Underneath the table sit 58 evidence-based micro-practices, each carrying its own contraindications. Depth is earned across sessions and resets on decompensation. That's a state machine, not a disposition.

The property this buys: the gate can't be talked out of. A client who is very articulate about why they're fine is only talking to the model. The gate reads the state score, never the message. That score is still a model's reading of their words, so how clearly someone writes can nudge it.

Crisis detection runs before generation, not after

The ordering is the design decision.

The common pattern is to filter after: the model writes something, a classifier checks it, you block or regenerate. That's fail-open. The model's output is the default and the filter is an exception — so every gap in the filter is a shipped reply.

Ours runs before any therapeutic reply is composed:

• Stage 1 — deterministic keyword pre-screen. Pure code, no model. Target ≤50ms, measured typically under 5ms. Cheap enough to sit in front of every message.
• Stage 2 — model classifier. A separate model call for anything the keywords don't catch. It reads the message and the last few messages before it, and the same message isn't guaranteed the same verdict. Target ≤3s; measured 0.8–2.0s on an earlier model.
• Stage 3 — continuous re-evaluation. Risk isn't assessed once and filed.

On trigger, generation is skipped entirely. A fixed message goes out — not a generated one. The named supervising clinician gets an email and a live console push, and a row lands in the alert table. The safety flag doesn't decay: a risk signal from three weeks ago is still in front of the model today.

The fixed message is deliberate. Acute risk is the last moment you want a language model being creative.

We're also deliberate about what does not trigger it: metaphor ("this is killing me"), reflective talk about past events, general sadness or burnout without stated intent, outward-directed anger. False positives erode trust faster than false negatives get caught downstream — a system that panics at sadness is one people stop talking to honestly.

What it cost

The honest part. Determinism isn't free, and most of the price is paid in places that don't show up in a demo.

Conversational flow. The gate sometimes produces a turn that's stiffer than the model "wanted." The model has a better-sounding sentence available and the table won't allow it. That's the trade working as designed, and it's still a real cost — you can feel it in transcripts.

Iteration speed. Adding a school isn't a prompt edit. It's a table change plus clinical review. The 58 micro-practices and their contraindications were authored, not few-shotted. A prompt-only competitor ships a new modality in an afternoon; we don't.

It can't do general conversation. By construction. The same machinery that steers it away from improvising a technique keeps it from being a good all-purpose chatbot. That's intentional, and it is nonetheless a capability we don't have.

Engineering surface. A deterministic spine is a lot of code that a prompt would have replaced. It has its own bugs, and they're our bugs rather than the model's.

What isn't solved

The crisis protocol catches explicit disclosure quickly, surfaces a hotline one tap away, and alerts the named clinician. Nobody is dispatched: no human enters the conversation, and any callback happens outside Hamo on the clinician's own initiative. It does not detect risk someone is deliberately concealing, and it can't intervene physically. What it buys is a number within reach and a clinician who knows — we don't claim more.

The table controls what goes in, not what comes out. The allowed moves reach the model as instructions. Code scans replies only for a short list of hard red lines, so a reply can still drift from its cell. The model can't change what a cell allows; it can only fail to follow it.

The outcome-slope metric — scoring whether a conversation moved someone toward equilibrium rather than just scoring each turn — is still roadmap, not shipped.

---

[Corrected 2026-11: the crisis pre-screen runs before scoring, not after it. Three steps use a model, the model picks from the moves Python allows, and the crisis classifier is not deterministic. The alert row is updated when a therapist acknowledges it.]

The full crisis protocol is published in detail so institutional partners can audit it rather than take our word for it. The nine methods and the shared spine are documented separately.