Operate
Failover and the first byte
We route between hosts of the same model, never between models. Before the first byte a retry is free. After it, nothing can move.
What failover is here
If the preferred host times out or errors before your response starts, the request moves to the next healthy host. You see a slower first token and nothing else: same model, same ID. We never substitute a different model, however similar or cheaper. The x-microrouter-provider header on every response names who actually served it.
The honest constraint: the first byte
Before the first content byte reaches you, a retry is free. Nothing has been delivered. Once the first byte is sent, provider switching is impossible. Splicing a second provider’s tokens into a half-delivered completion corrupts output, tool-call framing, and your client’s parser state.
| Event | Before first byte | After first byte |
|---|---|---|
| TTFT timeout | Abort, retry the next healthy route | Cannot occur. The first byte arrived. |
| Upstream 500 | Retry. You pay only for the successful attempt | Terminal error. What was produced is settled. |
| Fallback chain | Live. The full route list is available | Closed. The route is committed. |
You pay only for the attempt that succeeded. After the first byte, an upstream failure ends the stream with a terminal error. Under 50 output tokens, the charge is written off entirely.
Why TTFT is the health signal
Time to first token is the only latency measured while we can still act on it, so it drives route health. It is also the number published per upstream on every model page.
Errors on this path
request_timeoutNo first token from any upstream within the window. Safe to retry.upstream_errorThe upstream failed after streaming began, past the point of retry.model_unavailableEvery upstream failing. The error names working alternatives an agent can act on.