Reliability
Upstream worker and queue limits for LLM traffic
Set explicit capacity for each upstream origin, tenant, account, and work class.
Operators who need predictable capacity limits for provider HTTP calls.
The problem
A slow upstream origin can occupy capacity that other origins need. Media traffic also needs bounded capacity and reserved interactive capacity.
How LLM Proxy helps
LLM Proxy uses server.upstream_capacity to allocate capacity by origin. The scheduler applies server.upstream_rate_limits when each HTTP request starts.
How it works
- Set server.upstream_capacity.global ceilings for active and admitted HTTP requests.
- Declare active and queued capacity for each exact upstream origin.
- Set tenant, account, media, status, and transfer limits. Set an interactive reserve.
- Set server.upstream_rate_limits for the call budget of each upstream origin.
- Close each response body before the next poll. Handle HTTP 503 overload and HTTP 504 timeout responses.
Feature-to-benefit table
| Feature | Why it matters here | Example |
|---|---|---|
| Origin allocation | Each origin has an explicit admission allocation. | A slow origin cannot consume another origin's queue allocation. |
| Interactive reserve | Bulk media work leaves reserved capacity for interactive requests. | Text can proceed during media saturation. |
| Response ownership | An active request holds capacity until its response body closes. | Poll sleeps do not hold active HTTP capacity. |
Use-case examples
Local deployment
The configuration declares each local provider origin before startup.
Shared account
Tenants that use one saved connection share its account limit within each origin.
Operation telemetry
Admission events carry a request ID or an operation ID without credentials or prompts.
Objections and limitations
- Accepted durable jobs have separate capacity limits.
- Provider rate limits also apply.
- Transports that use the same exact origin share its capacity and rate budget.
FAQ
What is the main job of upstream worker queue limits?
LLM Proxy uses server.upstream_capacity to allocate capacity by origin. The scheduler applies server.upstream_rate_limits when each HTTP request starts.
Who should read this reliability resource?
Operators who need predictable capacity limits for provider HTTP calls.
Does this page claim provider performance or pricing advantages?
No. The supported claim is about LLM Proxy's documented routing, configuration, management, security, usage, and deployment contracts. Provider cost, speed, rankings, and benchmark claims are not made here.
Where should setup details come from?
Use the main README and implementation notes for current command, config, and endpoint details. This page summarizes the workflow without replacing LLM Proxy documentation.
What should I read next?
A closely related resource is OpenAI background response polling without client loops, which covers OpenAI background response polling.
Related resources
Use this pattern in LLM Proxy
Start from the canonical API reference, then use the management surface when the workflow needs tenant or provider configuration.
Open API reference