Reliability
Provider overload and timeout handling for LLM calls
Use clear 503 and 504 behavior for queue pressure and provider work that exceeds the proxy deadline.
Developers building retry and alerting behavior around LLM Proxy.
The problem
Failures are harder to handle when overload, provider timeout, missing credentials, and upstream errors collapse into one generic exception.
How LLM Proxy helps
LLM Proxy separates request queue pressure, disabled provider credentials, upstream provider rate limits, gateway timeout, and provider failures through documented status codes.
How it works
- Use 503 request queue full or provider not configured as service-availability signals.
- Use 504 as the overall proxy request deadline expiring.
- Use server.upstream_rate_limits to pace calls at actual upstream admission for each configured origin.
- Use 429 for upstream provider rate limits.
- Use 502 for other upstream provider API failures.
Feature-to-benefit table
| Feature | Why it matters here | Example |
|---|---|---|
| Queue status | The shared upstream operation queue returns service-unavailable behavior when full. | Callers can back off before adding more pressure. |
| Deadline status | Long requests that exceed their accepted proxy work budget return a canonical 504. | Clients should not poll llm-proxy after a timeout. |
| Provider error mapping | Rate limits and provider failures map to distinct statuses. | Retry logic can be status-aware. |
Use-case examples
Capacity alert
A spike in queue-full responses points to proxy capacity tuning.
Long prompt timeout
A semantic-review request returns 504 when provider work exceeds the configured deadline.
Disabled provider
A selected provider without an API key returns 503 provider not configured.
Objections and limitations
- Status codes do not guarantee safe automatic retries for every prompt.
- Provider-side behavior may still vary by upstream service.
- Waiting for a server.upstream_rate_limits slot remains bounded by the request deadline and can therefore end as 504.
FAQ
What is the main job of LLM provider overload timeout handling?
LLM Proxy separates request queue pressure, disabled provider credentials, upstream provider rate limits, gateway timeout, and provider failures through documented status codes.
Who should read this reliability resource?
Developers building retry and alerting behavior around LLM Proxy.
Does this page claim provider performance or pricing advantages?
No. The supported claim is about LLM Proxy's documented routing, configuration, management, security, usage, and deployment contracts. Provider cost, speed, rankings, and benchmark claims are not made here.
Where should setup details come from?
Use the main README and implementation notes for current command, config, and endpoint details. This page summarizes the workflow without replacing LLM Proxy documentation.
What should I read next?
A closely related resource is OpenAI background response polling without client loops, which covers OpenAI background response polling.
Related resources
Use this pattern in LLM Proxy
Start from the canonical API reference, then use the management surface when the workflow needs tenant or provider configuration.
Open API reference