Streaming

Set stream: true on a chat request to receive tokens as they are generated. Roadie streams Server-Sent Events over the POST response (Content-Type: text/event-stream) — the same transport OpenAI and Anthropic use, so every HTTP intermediary and fetch client already supports it. With the SDK you get an async iterable:

for await (const event of roadie.chat.stream({ model: 'openai/gpt-4o', messages })) {
  if (event.type === 'content_delta') process.stdout.write(event.delta);
  if (event.type === 'message_end') console.log(event.usage, event.cost);
}

Streaming applies to chat only. Embeddings and image requests return a single response.

Framing

SSE (default) — each event is a data: {json} line; the terminal sentinel is data: [DONE].
NDJSON — request Accept: application/x-ndjson (SDK: { framing: 'ndjson' }) for one JSON object per line, no data: prefix and no [DONE] (end-of-stream is the signal). Same event objects, different framing.

Buffering is disabled at the edge (Cache-Control: no-store, X-Accel-Buffering: no) so frames reach the client as they are produced.

Unified stream events

Each frame carries a type:

`type`	Fields	Meaning
`message_start`	`providerRequestId?`	The stream has opened.
`content_delta`	`delta`	An incremental chunk of assistant text.
`tool_call_delta`	`index`, `id?`, `name?`, `argumentsDelta`	An incremental tool-call fragment.
`message_end`	`finish_reason?`, `usage?`, `cost?`	Terminal success frame — final usage and estimated cost.
`error`	`error: { type, code, message, request_id }`	Terminal error frame for a mid-stream failure; nothing follows it.

Concatenate the content_delta deltas to reconstruct the full text. The message_end frame is where the final token usage and estimated cost arrive.

These frames intentionally mix casing: message_start / content_delta / tool_call_delta carry camelCase fields (providerRequestId, argumentsDelta), while message_end uses the snake_case wire shapes (finish_reason, usage, cost). The SDK yields them verbatim.

Heartbeats

The gateway emits an SSE comment heartbeat (a line starting with :) roughly every 15 seconds so mobile radios, NATs, and proxies don’t drop an idle connection during long tool-call pauses. Ignore comment lines when parsing by hand; the SDK handles them for you.

Commit point — no fallback after the first byte

The stream commits on its first event: once the client has seen bytes, the request can no longer fall back or retry a different target. A pre-first-byte failure is still eligible for fallback; a mid-stream drop finalizes as a terminal error frame (provider_stream_interrupted), never a silent switch to another provider.

Cancellation and timeouts

Pass an AbortSignal to stop iteration and close the connection:

const controller = new AbortController();
const stream = roadie.chat.stream({ model: 'openai/gpt-4o', messages }, { signal: controller.signal });
// later: controller.abort();

A per-request timeoutMs composes with your signal; a fired deadline throws APIConnectionTimeoutError. You can also cancel an in-flight request server-side with POST /v1/requests/{request_id}/cancel.

How usage is finalized

Usage is metered in exactly one place — the finalizer — so it is recorded no matter how a stream ends:

Terminal state	Usage recorded	Outcome
Stream completes	Provider-reported, exact	`ok`
Client disconnects mid-stream	Input + tokens streamed so far (estimated if no provider usage arrived)	`client_disconnected` — still billable
Provider disconnects mid-stream	Tokens received so far, estimated	`provider_stream_interrupted` (you receive a terminal `error` frame)
Request timeout	As above	`timeout`

This is why a disconnected stream still meters what it produced: the tokens were generated.

Get started

Core concepts

Guides

SDKs

Framing

Unified stream events

Heartbeats

Commit point — no fallback after the first byte

Cancellation and timeouts

How usage is finalized

​Framing

​Unified stream events

​Heartbeats

​Commit point — no fallback after the first byte

​Cancellation and timeouts

​How usage is finalized

Framing

Unified stream events

Heartbeats

Commit point — no fallback after the first byte

Cancellation and timeouts

How usage is finalized