Architecture

Four layers, each with one job. The boundaries are the point: they are what makes the system auditable.

The layers

Routes never touch Pinecone or a raw HTTP client. They call a service. This is not stylistic — it is what keeps a permission check from being skippable.

A planned turn, and one bounded loop behind it

Every turn is planned first. plan_turn makes one structured call on the gather model and merges it with member-cue rules ("my", "our", "am I", mera/meri, मेरा/मेरी) that can only move a turn towards the member's own data. The plan says whether the turn is a member question, a mixed one, a general one, small talk or an action, which parts of the member's record matter, and the searches to run. Code then chooses the reads, and the original bounded gather loop is kept for actions and for turns the router could not plan.

A signed-out member question pauses at auth_gate instead of being answered without the member's data; POST /chat/resume carries it on after sign-in. A signed-out mixed question answers its general part first, labelled as general, and then pauses.

START
  │
  ▼
prepare ──────────────────────────────────► escalate ──► END
  │
  ▼
plan_turn ─┬─ member, signed out ──► auth_gate ⇢ /chat/resume ──┐
           ├─ member / mixed / knowledge ──► prefetch ◄─────────┘
           │                                   │ ▲ one follow-up
           │                                   ▼ │
           ├─ action / no plan ──► gather ◄─► execute_tools ──► confirm
           │                         │             │
           └─ small talk ────────────┴─────────────┴──► build_context
                                                            │
                     mixed, signed out: auth_gate ◄── finalize ◄── answer

Both prepare and gather can divert to escalate. Every other path converges on build_context, so a turn always ends in an answer or a hand-over.

The loop's exit condition is read from state, never asked of the model: spent tool steps, a spent model-call budget, or an assistant message with no tool calls. budget_exhausted exists because every outstanding tool_call needs a reply before the next provider request, so the notice is delivered as those replies rather than as prose.

Two model roles

The split is what makes a single loop safe to stream.

A single model that produced text and then decided to call a tool would already have leaked half an answer. Three independent guards prevent it: the gather model is constructed with disable_streaming, so it cannot emit chunks at all; its nostream tag makes LangGraph drop its messages from the message stream; and the runner only forwards tokens whose node is answer. Any one would do. All three is the point.

The authorisation boundary

Tools are not dispatched by hand. execute_tools is LangGraph's ToolNode, constructed with an async interceptor:

ToolNode(
    as_langchain_tools(resolved),
    awrap_tool_call=authorise_and_execute,
    handle_tool_errors=False,
    name=NODE_NAME,
)

awrap_tool_call is the only path a tool call can take, which is what makes the five-step chain in Tool authorisation unskippable. handle_tool_errors=False matters as much: the interceptor maps every failure onto a stable code itself, instead of letting the framework flatten it into an untyped string that neither the event contract nor the audit table can read.

The node is built with every registered tool, not just the caller's, so a call to a forbidden tool reaches the interceptor to be refused and audited rather than vanishing.

Auth never reaches the checkpoint

The authenticated caller travels as LangGraph runtime context — a frozen dataclass passed at invoke time — not as graph state.

@dataclass(frozen=True)
class ChatRuntimeContext:
    auth: AuthContext
    request_id: str
    conversation_id: UUID
    channel: str
    settings: Settings
    rag: RagService

Runtime context is per-run and is not checkpointed, so the AuthContext cannot reach Postgres, a prompt or a trace. That turns "no credentials in persisted state" into a property of the design rather than a rule somebody has to remember. The context carries no token in the first place: AuthContext holds an opaque credential_ref, which the executor exchanges at call time.

State, and where it lives

The process holds no conversation memory. The thread does, in Postgres.

History is still accepted on the wire and still capped, but the server is now authoritative: a thread with no checkpoint is seeded from history plus the new message, and a thread that already has one takes only the new message.

{
  "conversation_id": "web-V1StGXR8Z5",
  "channel": "web",
  "message": { "role": "user", "content": "Why is my claim still pending?" },
  "history": [{ "role": "assistant", "content": "…" }],
  "client": { "locale": "en-IN", "timezone": "Asia/Kolkata" }
}

The confirmation pause

A write or a send never runs in the same pass as anything else. The interceptor parks it as a pending_action with an "awaiting confirmation" reply, and the confirm node calls interrupt() before any side effect — resuming re-runs the node from the top, so anything executed before the pause would run again on every resume.

The pause is visible on the wire as action.confirmation_required, and the run is resumed by POST /chat/confirm. The confirmation row is the authority for who may answer: it records the member who was asked and when the request expires, so an approval survives a process restart and cannot be given by somebody else.

Untrusted content

Retrieved documents and tool results are data, never instructions. The context builder and the sanitizer fence both in the delimiters the system prompt names, so a document containing something shaped like a prompt is read as a document containing something shaped like a prompt.

The builder is plain Python with no model call and no I/O. Given the same state it produces byte-identical messages, which is what makes an answer reproducible when something goes wrong. It owns ordering (fixed, never scored), deduplication, the evidence and history budgets, and that fence.