Streaming

Answers stream. A grounded insurance answer takes several seconds to assemble — retrieval, one or more tool calls, then generation — and a member watching a blank screen assumes it has failed.

The backend exposes a stable event format rather than passing provider events through. OpenRouter, LangChain and LangGraph are implementation details; the contract below is not.

Transport

Three endpoints, one contract. All take Accept: text/event-stream and reply text/event-stream, where every frame carries a named event and a JSON payload.

Responses are sent with Cache-Control: no-cache, no-transform and X-Accel-Buffering: no, because nginx buffers proxied responses by default and would otherwise deliver a whole answer in one lump at the end.

Authentication

Every request carries two headers, for two different principals:

API keys are server-side secrets. Browsers and apps call through their own backend, which holds the key. A pcai_live_ key works only in production and a pcai_test_ key only outside it. Each key has a per-minute request limit.

Without X-PlanCover-Session, /chat/stream runs as an anonymous, ephemeral member: knowledge questions only, and no conversation continuity, because each call gets a fresh identity. To keep a thread, mint a session once with POST /auth/session and send it on every call. /chat/confirm, /chat/resume and every /auth/* route except /auth/session need the session, and answer 401 without it.

The member session used to be the Authorization bearer. It is not any more: a session id sent as the bearer is refused as an invalid API key.

Refusals are HTTP status codes

Every check that can refuse a turn runs before the stream opens: the API key and its limit, the member's quota, the conversation, and a confirmation's ownership and expiry. A refusal is therefore an ordinary JSON response with a real status and the error envelope, never an error frame on a 200:

Only a failure after the first byte is an error event.

ChatStreamRequest.client carries hints, never authority: locale, timezone and rerank. rerank is a test client's per-turn reranker switch: false skips the configured reranker for that turn, true or absent keeps it, and true cannot enable a reranker that is not configured. Only a non-prod deployment honours it; prod ignores it. Each search's funnel reports what became of it.

Events

message.started

Opens the stream and names the message. model is the answer model — the one whose tokens the member will actually read.

{ "message_id": "0f3c…", "conversation_id": "9b2e…", "model": "openai/gpt-4o-mini" }

retrieval.completed

Emitted once, after the loop, when the evidence for this answer is fixed and before generation begins — so the client is told what actually grounded the answer rather than once per retrieval call. A turn that searched emits it even when nothing came back or the search failed, with documents: []; a turn that never searched emits no frame at all. duration_ms is the turn's total time in knowledge searches.

{
  "query": "day care coverage",
  "duration_ms": 312,
  "documents": [
    {
      "index": 1,
      "id": "chunk-gmc-14",
      "title": "Group Medical Cover — Day-care procedures — p.12",
      "document_name": "Group Medical Cover",
      "section": "Day-care procedures",
      "page": 12,
      "score": 0.91,
      "snippet": "…",
      "source": "policy_wording"
    }
  ],
  "searches": [
    {
      "tool_call_id": "call_1",
      "query": "day care coverage",
      "status": "ok",
      "duration_ms": 312,
      "funnel": {
        "generation": 12,
        "returned": 40,
        "dropped_floor": 31,
        "dropped_slice": 3,
        "dropped_dedupe": 0,
        "dropped_budget": 0,
        "kept": 6,
        "rerank": "ok",
        "rerank_hint": true
      }
    }
  ]
}

query is what was searched — the query the model wrote, not the member's message — with several searches joined by |. searches has one entry per knowledge search, in the order they ran. funnel is the same set of counts as the retrieval_searched log line: the candidate pool, what the relevance floor, the chunk cap, deduplication and the character budget each dropped, and what was kept. rerank is ok; off when no reranker is configured; fallback when it failed and the dense order stood, with no relevance floor; or skipped_by_client when the request's client.rerank: false skipped it, also with no floor. rerank_hint is that hint as the search applied it, absent when none was sent or the deployment ignored it, so "off" beside true means the client asked for a reranker the deployment does not have. A failed search (status: "error") has no funnel. The turn then merges every search's kept passages, so documents can be shorter than the sum of kept.

tool.started / tool.completed

A pair per executor decision, sharing a tool_call_id — including refusals, which complete immediately with duration_ms: 0. Arguments are the redacted preview and never contain credentials: the model proposes business arguments only, and credential_ref is exchanged for a token inside the service.

{ "tool_call_id": "call_1", "name": "get_plan_details", "arguments": { "portal_ref": "100001" } }

{
  "tool_call_id": "call_1",
  "name": "get_plan_details",
  "status": "ok",
  "duration_ms": 486,
  "result": { "suminsured": "500000", "title": "Group Medical Cover" }
}

status is ok or error. A failure carries error — a human-readable message, not a code — and no result. The stable failure code is not on the wire: it reaches the answer model inside the tool message, so a client cannot branch on it.

action.confirmation_required

The turn has paused. A write or a send was proposed, nothing upstream has happened, and nothing may claim it has. Emitted once per pause, by the runner rather than by a node, so it does not re-fire on resume.

{
  "confirmation_id": "3f9a…",
  "tool_call_id": "call_2",
  "name": "request_callback",
  "summary": "Ask a PlanCover advisor to call you back about: my claim is stuck",
  "arguments_preview": { "reason": "my claim is stuck" },
  "risk": "send",
  "expires_at": "2026-09-12T11:24:07+00:00"
}

arguments_preview is the redacted copy, not what will run. The two differ whenever a member has typed something personal into a free-text field.

action.completed

How a confirmed action ended, emitted on the resumed stream.

{ "confirmation_id": "3f9a…", "status": "ok", "decision": "approve" }

status is a closed set of three. The audit trail distinguishes why an action did not run — rejected, expired, tool_failed all appear in tool_audit — but the wire only separates "the member said no" from "it did not happen", with the reason in error. Widening the wire vocabulary would mean every client had to learn a new value each time the backend grew a new way to decline.

auth.required

The member's PlanCover data is needed and they are not signed in. This is the signal behind the in-chat sign-in card, and it comes in two shapes.

Resumable — a planned member question, paused before anything was read:

{
  "reason": "sign_in_required",
  "tool": "get_covered_benefits",
  "message": "To answer that from your own policy, I need you to sign in to PlanCover. …",
  "resumable": true,
  "resume_id": "78d8933d80fd576aaa549d0de2bee768"
}

The turn is paused on an interrupt and the stream ends with finish_reason: "awaiting_sign_in". A member-only question gets the same fixed text as token.delta frames first, in the member's language, so the card sits under "sign in below"; a mixed question has already streamed its general part, labelled as general, and gets no extra text. Emitted once per pause, by the runner, so it does not re-fire on resume. After sign-in, POST /chat/resume carries the same turn on — see Resuming after sign-in.

Not resumable — the older shape, without resumable or resume_id. A tool on the gather path (an action turn, or one the router could not plan) refused for want of a credential. That turn finished; the client asks the question again once signed in. Older clients only know this shape, and treating every card this way still works: a re-sent question on a paused thread starts a fresh run.

reason is the field a client can rely on. tool names a tool for the event tape. message is fixed text for the member on the resumable shape and a note for the answer model on the other, so a client renders its own sign-in copy.

Neither shape ever carries a mobile number, an OTP, an identifier or the question. Signing in happens entirely outside the graph over POST /auth/login and POST /auth/verify, because interrupt values and resume values are both checkpointed. The credential attaches to the same session id, so the conversation is never re-keyed. On WhatsApp, which has no card, a member question is answered with a fixed "sign in through the PlanCover app" text and finish_reason: "sign_in_required" instead.

token.delta

The answer text, in order. Hundreds of these frames make up one answer. Only the answer node's model output is forwarded, so the gather model's reasoning can never reach the member. The one exception is escalate, which writes its canned hand-over reply as a token.delta directly — it is the turn's only text and it comes from a constant rather than a provider.

{ "text": "Your claim " }

citation

One per source the answer actually rests on, emitted after the text. A citation that does not correspond to evidence selected for this turn is dropped rather than sent: a fabricated source is worse than no source, because it looks like grounding.

{ "id": "chunk-gmc-14", "title": "Group Medical Cover — Day-care procedures — p.12", "document_id": "gmc-2026", "snippet": "…" }

url is present only when a document base URL is configured. It is attached here, at emit time, from a document_id that travelled alone — document URLs never enter graph state, a prompt or a trace.

message.completed

Closes the stream with the finish reason and, when the provider reported it, usage.

{ "message_id": "0f3c…", "finish_reason": "stop", "usage": { "prompt_tokens": 1841, "completion_tokens": 212, "total_tokens": 2053 } }

The finish reason is read off the graph's own channel, so it says what the graph actually did rather than merely that the stream ended.

Treat it as an open set: a reason added later should land in the client's event tape rather than fail validation.

A client that treats awaiting_confirmation as an ending will leave a pending action on screen forever. It is a pause, and only POST /chat/confirm moves it. insufficient_evidence is a signal, not a rewrite — the answer is streamed as written, and flagged rather than silently edited, because hiding a grounding failure is worse than showing one.

error

A failure the member needs to know about, once the stream has opened. Carries a code, a human-readable message, and whether retrying is worthwhile. Terminates the stream. Refusals before the stream opens are HTTP statuses instead (see above).

{ "code": "upstream_timeout", "message": "…", "retryable": true }

Confirming an action

POST /chat/confirm answers a pending action.confirmation_required and continues the same turn on the same thread. It returns the same event contract: the stream carries on from where the pause left it, so the reply the member reads is a normal token.delta sequence.

{
  "conversation_id": "web-V1StGXR8Z5",
  "confirmation_id": "3f9a…",
  "decision": "approve",
  "feedback": null
}

The confirmations row is the authority, not the request: it records who was asked, and the endpoint refuses a confirmation that was already answered or has expired, with a real HTTP 403 before any stream opens. One that belongs to somebody else gets the same answer as one that never existed, so the endpoint cannot be used to discover that a confirmation is out there. An unreadable resume value is treated as a rejection. Nothing about a write fails open.

Because a confirmed write must not be replayable, the resumed run commits its checkpoint synchronously rather than in the background.

Resuming after sign-in

POST /chat/resume carries on a turn that ended with a resumable auth.required, once the member has signed in on the same session. The reads the turn planned then run (and, for a mixed question, its searches again), and the answer streams on the same event contract.

{ "conversation_id": "web-V1StGXR8Z5" }

Nothing else is sent: the question is already in the thread, and the credential is the caller's own stored one, which the resumed run reads afresh along with the portals it grants. The conversation is looked up by the caller and its own name and never created. Every check runs before the stream opens, so a refusal is a real HTTP 403 with one message for every case — unknown conversation, somebody else's, still signed out, or not waiting — and the endpoint cannot be used to learn which conversations exist. It counts against the message quota (429), and the resumed run commits its checkpoints synchronously, like a confirmation.

A new message instead of a resume starts a fresh run, and the paused one is dropped; the fixed sign-in text the member saw is stored first, so the history the model reads matches what they saw.

Ordering guarantees

All three are asserted end to end in tests/test_chat_api.py, against real requests through the route rather than against the event generator alone.

A stream that ends without a terminator is treated as a failure by the client, not as a completed answer. If the graph raises past the runner or the generator is closed early, the SSE layer emits stream_interrupted rather than letting the client guess.

Heartbeats

While the graph is thinking — a slow tool, a slow provider — the stream would otherwise go quiet long enough for an idle proxy to close it. A comment frame goes out every SSE_HEARTBEAT_SECONDS:

: keep-alive

It is a comment, not a named event, so a client ignores it without decoding anything. The heartbeat is a race between the next event and a timer rather than a background task, so it cannot outlive the stream or fire after it closed.

Cancellation

Disconnects propagate, in two stages.