Streaming

Stream tokens as they're generated, identically to the OpenAI SDK.

Streaming works exactly as in the OpenAI SDK: pass stream: true and iterate the async iterable. The SDK changes nothing about the shape of the stream.

const stream = await client.chat.completions.create({
  model: "nova-lite",
  messages: [{ role: "user", content: "Write a haiku." }],
  stream: true,
});
 
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Each chunk is a standard OpenAI chat.completion.chunk. The return type is fully typed: stream: true yields a stream, stream: false (or omitted) yields a single completion, the same overloads as upstream.

Billing on streams

Output length isn't known until the stream ends, so Fleeex reserves a ceiling up front and settles the real cost when the stream finishes. The wallet can never go negative. From the SDK's side there's nothing to do: you just iterate.

To learn what the stream actually cost, add stream_options: { include_usage: true }. See Streaming usage.

Errors happen before the first chunk

Every refusal is decided before a single SSE byte is written, so it surfaces when you await the create call, never mid-iteration. That covers the 402, a 400 on an unsupported parameter, the 403s, and the stream limit below.

import { PaymentRequiredError } from "@fleeex/sdk";
 
try {
  const stream = await client.chat.completions.create({
    model: "nova-lite",
    messages,
    stream: true,
  });
  for await (const chunk of stream) {
    /* … */
  }
} catch (err) {
  if (err instanceof PaymentRequiredError) {
    if (err.topupUrl) redirect(err.topupUrl);
  } else {
    throw err;
  }
}

Once the headers are committed the response can no longer become an HTTP error: a provider failure mid-stream ends the stream instead, and you are billed for what was delivered. So handle a stream that stops short of a finish_reason as a partial answer rather than as a success.

See Balance and payments for the full payment flow.

A stream can be cut, and says so

Two ceilings can end a stream early. Both look the same on the wire, finish_reason: "length", and both settle honestly: you are charged for what was delivered, and the rest of the reservation is released.

• The cost ceiling. Output length isn't known up front, so if the running cost would overrun the reservation the stream stops rather than letting the wallet go negative. A larger max_completion_tokens reserves more, so it also cuts later, at the price of holding more funds. See max_completion_tokens.
• A wall-clock ceiling, for a stream that trickles forever.

So finish_reason: "length" means the answer was truncated, by your max_completion_tokens or by one of those two ceilings. It never means the model finished on its own. The three are not distinguishable, so treat it as "there was more to say". If you need to know what a cut stream consumed, ask for the usage chunk.

One user, several streams

Concurrent streams are capped per end user, so one user can't hold every slot. Over the cap, create rejects with a 429 CONCURRENT_STREAM_LIMIT carrying a Retry-After header.

It's refused before any reservation or provider call, so it costs nothing, and it clears as soon as one of that user's own streams ends. The remedy is fewer streams in flight, not a slower request rate.