Why Streaming Matters for LLM Applications
A non-streaming LLM response forces users to wait in silence while the model generates its full output — often 3–10 seconds for a typical response. Streaming delivers tokens as they are generated, so the first words appear within 200–500 milliseconds and the response builds progressively on screen. This transforms the perceived experience from waiting for a slow system to watching a fast thinker. Research on perceived responsiveness consistently shows that users rate streaming interfaces significantly higher than equivalent non-streaming interfaces, even when total generation time is identical. For any customer-facing LLM application, streaming is not a nice-to-have — it is the expected baseline.
How LLM Streaming Works
LLM providers stream responses using Server-Sent Events (SSE) — a simple HTTP protocol where the server sends newline-delimited event data over a persistent connection. Each event contains one or more generated tokens. The client reads events as they arrive and renders them progressively. The infrastructure is simple: a single long-lived HTTP connection, standard HTTP/1.1 chunked transfer encoding, and a straightforward event parsing loop on the client. No WebSockets, no polling, no special browser APIs beyond the standard EventSource or fetch with streaming body.
Streaming from Anthropic Claude
import anthropic
client = anthropic.Anthropic()
# Basic streaming — prints tokens as they arrive
with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": "Explain how transformers work in detail."}]
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
print() # newline after stream completes
# Access full response metadata after streaming
final = stream.get_final_message()
print(f"
Tokens: {final.usage.input_tokens} in / {final.usage.output_tokens} out")Streaming from OpenAI
from openai import OpenAI
client = OpenAI()
stream = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Write a Python function to parse JSON safely."}],
stream=True
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print()Building a Streaming FastAPI Endpoint
FastAPI’s StreamingResponse with text/event-stream media type is the standard way to serve streaming LLM responses over HTTP:
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from fastapi.middleware.cors import CORSMiddleware
import anthropic
import json
app = FastAPI()
# Allow browser clients to connect
app.add_middleware(CORSMiddleware, allow_origins=["*"],
allow_methods=["*"], allow_headers=["*"])
client = anthropic.Anthropic()
@app.post("/chat/stream")
async def stream_chat(request: dict):
user_message = request.get("message", "")
system = request.get("system", "You are a helpful assistant.")
async def generate():
with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=1024,
system=system,
messages=[{"role": "user", "content": user_message}]
) as stream:
for text in stream.text_stream:
# SSE format: "data: {payload}
"
yield f"data: {json.dumps({'text': text})}
"
yield "data: [DONE]
"
return StreamingResponse(generate(), media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)The X-Accel-Buffering: no header is critical when running behind Nginx — without it, Nginx buffers the response and the client sees nothing until the stream is complete, defeating the purpose of streaming entirely.
Consuming the Stream in JavaScript
async function streamChat(message) {
const response = await fetch('/chat/stream', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({message})
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const {done, value} = await reader.read();
if (done) break;
buffer += decoder.decode(value, {stream: true});
const lines = buffer.split('
');
buffer = lines.pop(); // keep incomplete event in buffer
for (const line of lines) {
if (line.startsWith('data: ')) {
const data = line.slice(6);
if (data === '[DONE]') return;
try {
const {text} = JSON.parse(data);
document.getElementById('output').textContent += text;
} catch(e) {}
}
}
}
}
// Usage
streamChat("Explain async/await in JavaScript.");Async Streaming with AsyncAnthropic
For true async FastAPI performance — handling multiple concurrent streaming requests efficiently — use the async client:
from anthropic import AsyncAnthropic
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio, json
app = FastAPI()
client = AsyncAnthropic()
@app.post("/chat/stream")
async def stream_chat(request: dict):
async def generate():
async with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role":"user","content":request["message"]}]
) as stream:
async for text in stream.text_stream:
yield f"data: {json.dumps({'text': text})}
"
yield "data: [DONE]
"
return StreamingResponse(generate(), media_type="text/event-stream",
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no"})The async client allows FastAPI to handle other requests while waiting for LLM tokens, rather than blocking the event loop. For a server handling 50 concurrent streaming requests, this difference between sync and async is the difference between responsive and unresponsive for all users when any single request is slow.
Streaming with Conversation History
from pydantic import BaseModel
class ChatRequest(BaseModel):
messages: list[dict] # Full conversation history
system: str = "You are a helpful assistant."
max_tokens: int = 1024
@app.post("/chat/stream")
async def stream_chat(request: ChatRequest):
async def generate():
async with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=request.max_tokens,
system=request.system,
messages=request.messages
) as stream:
async for text in stream.text_stream:
yield f"data: {json.dumps({'text': text})}
"
# Send final message metadata for token counting
final = await stream.get_final_message()
yield f"data: {json.dumps({'done': True, 'usage': {'input': final.usage.input_tokens, 'output': final.usage.output_tokens}})}
"
return StreamingResponse(generate(), media_type="text/event-stream",
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no"})Handling Errors in Streaming Responses
When an error occurs mid-stream, you cannot change the HTTP status code — it was already sent as 200 when the stream started. Communicate errors via the stream itself:
@app.post("/chat/stream")
async def stream_chat(request: dict):
async def generate():
try:
async with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role":"user","content":request.get("message","")}]
) as stream:
async for text in stream.text_stream:
yield f"data: {json.dumps({'text': text})}
"
except anthropic.RateLimitError:
yield f"data: {json.dumps({'error': 'rate_limit', 'message': 'Too many requests. Please try again shortly.'})}
"
except anthropic.APIError as e:
yield f"data: {json.dumps({'error': 'api_error', 'message': str(e)})}
"
finally:
yield "data: [DONE]
"
return StreamingResponse(generate(), media_type="text/event-stream",
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no"})Always send [DONE] in the finally block so the client knows the stream ended regardless of whether an error occurred. Client-side code that only looks for [DONE] and never receives it will hang indefinitely waiting for more data.
Streaming Behind Nginx and Load Balancers
Several infrastructure components can accidentally buffer streaming responses. For Nginx, set these directives on your location block:
location /chat/stream {
proxy_pass http://backend;
proxy_buffering off; # Disable response buffering
proxy_cache off; # Disable caching
proxy_set_header Connection '';
proxy_http_version 1.1; # Required for chunked transfer
chunked_transfer_encoding on;
proxy_read_timeout 300s; # Long timeout for slow generation
}AWS Application Load Balancers buffer streaming responses by default — enable HTTP/2 on your target group and set idle timeout to 300 seconds. Cloudflare’s free and pro plans buffer SSE streams; disable “Rocket Loader” and set Response Buffering off in Cloudflare’s configuration. These buffering issues all manifest the same way: the client receives the full response at once after a long wait, rather than progressively. If you encounter this symptom, work through the infrastructure stack from the client backwards to find the buffering component.
Streaming Latency Optimisation
Time-to-first-token (TTFT) — the delay between sending the request and receiving the first character — is the most important latency metric for streaming applications. Several factors affect it. Model size matters: smaller models (Haiku, Flash, GPT-4o mini) produce first tokens significantly faster than frontier models. Prompt length matters: longer input prompts take longer to process before generation begins — prompt caching reduces this for shared prefixes. Provider infrastructure matters: some providers offer dedicated capacity tiers with lower TTFT under load. Measure TTFT separately from generation speed in your observability stack and optimise them independently. For interactive applications, a 300ms TTFT with 20 tok/s generation feels much faster than a 2000ms TTFT with 80 tok/s generation, even though the full response arrives at similar times — the initial responsiveness drives user perception far more than raw throughput.
Rate Limiting Streaming Endpoints
Streaming endpoints are more expensive to rate-limit correctly than regular endpoints because each streaming request holds an open connection for its duration. Standard request-count rate limiting undercounts the load — a user making 5 concurrent 30-second streaming requests is far more expensive than 5 instantaneous requests. Rate limit streaming endpoints on two dimensions: concurrent connections per user (typically 2–3 for interactive chat applications) and tokens generated per minute. Track active streaming connections per user ID in Redis and reject new requests when the limit is exceeded. Token-per-minute tracking uses the usage metadata from the final message event to record actual tokens consumed, enabling fair billing and abuse prevention.
Testing Streaming Endpoints
Testing streaming APIs requires a slightly different approach than testing standard REST endpoints. Use httpx for Python-based streaming tests:
import httpx, json, pytest
@pytest.mark.asyncio
async def test_streaming_endpoint():
async with httpx.AsyncClient(base_url="http://localhost:8000") as client:
async with client.stream("POST", "/chat/stream",
json={"message": "Say hello in exactly 3 words."}) as response:
assert response.status_code == 200
assert "text/event-stream" in response.headers["content-type"]
tokens = []
done = False
async for line in response.aiter_lines():
if line.startswith("data: "):
data = line[6:]
if data == "[DONE]":
done = True
break
payload = json.loads(data)
if "text" in payload:
tokens.append(payload["text"])
assert done, "Stream did not send [DONE]"
full_response = "".join(tokens)
assert len(full_response) > 0, "Empty response"React Integration: Displaying Streamed Tokens
import { useState } from 'react';
function StreamingChat() {
const [response, setResponse] = useState('');
const [loading, setLoading] = useState(false);
const sendMessage = async (message) => {
setLoading(true);
setResponse('');
const res = await fetch('/chat/stream', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({message})
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const {done, value} = await reader.read();
if (done) break;
buffer += decoder.decode(value, {stream: true});
const events = buffer.split('
');
buffer = events.pop();
for (const event of events) {
if (event.startsWith('data: ')) {
const data = event.slice(6);
if (data === '[DONE]') { setLoading(false); return; }
try {
const {text} = JSON.parse(data);
setResponse(prev => prev + text);
} catch(e) {}
}
}
}
setLoading(false);
};
return (
<div>
<button onClick={() => sendMessage("Explain React hooks.")} disabled={loading}>
{loading ? 'Generating...' : 'Ask'}
</button>
<pre>{response}</pre>
</div>
);
}State updates in React are batched, so appending tokens one at a time via setResponse(prev => prev + text) is efficient — React batches the re-renders automatically. For high-speed streaming (100+ tok/s), consider batching token updates with a small timeout to avoid thrashing the render cycle, though for typical LLM streaming speeds (20–80 tok/s) token-by-token updates render smoothly without any batching.
Streaming vs. Non-Streaming: When to Use Each
Streaming is not always the right choice. For user-facing conversational interfaces, it is almost always better — the perceived responsiveness improvement is significant and straightforward to implement. For background processing pipelines where results are not displayed to a user in real time, non-streaming is simpler and sufficient. For structured output extraction where you need the complete JSON before processing it, streaming is incompatible — you cannot parse partial JSON. For applications using the batch API for offline workloads, streaming is unavailable by design. The rule of thumb: if a human is waiting for the response to appear on screen, stream it. If the response is going to a database, a queue, or an automated processing step, streaming adds complexity without benefit.
Security Considerations for Streaming Endpoints
Streaming endpoints have a few security considerations worth addressing explicitly. Authentication must be checked before the stream starts — once the StreamingResponse is returned, you cannot reject the request mid-stream. Validate all inputs and authenticate the user in the endpoint function body before returning the StreamingResponse. Set a maximum generation length via max_tokens to prevent runaway responses that keep connections open indefinitely. Log the start and end of each streaming session with user ID and token counts — streaming requests are harder to audit than standard requests because they do not appear as single complete entries in access logs. And implement connection timeouts at the application layer: if a streaming response takes longer than your maximum expected generation time, close the connection and log the anomaly.