Reliability
Upstream worker and queue limits for LLM traffic
Use shared worker and queue controls to bound upstream HTTP operations for text and dictation.
Operators who need predictable capacity limits for provider HTTP calls.
The problem
Unlimited upstream calls can exhaust provider quotas or local resources, while long OpenAI polling sleeps should not occupy scarce worker capacity.
How LLM Proxy helps
LLM Proxy combines server.workers and server.queue_size concurrency bounds with server.upstream_rate_limits rolling-window rules applied at actual upstream admission.
How it works
- Set server.workers for active upstream HTTP concurrency.
- Set server.queue_size for pending upstream operations.
- Set server.upstream_rate_limits rules for strict call budgets keyed by normalized upstream origin.
- Let OpenAI background poll sleeps release worker capacity between polls.
- Handle 503 request queue full and 504 timeout responses at the caller boundary.
Feature-to-benefit table
| Feature | Why it matters here | Example |
|---|---|---|
| Shared limiter | Text providers and dictation use the same upstream HTTP operation limit. | Capacity policy is centralized. |
| Queue pressure signal | A full queue returns service-unavailable behavior. | Callers can distinguish overload from provider failure. |
| Polling separation | OpenAI poll sleeps do not occupy worker slots. | Other requests can proceed while a background response waits. |
Use-case examples
Low-capacity environment
A local deployment sets one worker and a small queue to keep provider traffic controlled.
Mixed text and audio
Dictation and text calls share the same upstream HTTP admission boundary.
Operational debugging
A queue-full status points to proxy-side capacity, not a malformed request.
Objections and limitations
- These controls limit upstream HTTP operations, not the number of connected client requests.
- They do not replace provider-side rate limits.
- Rate rules are keyed by exact normalized HTTP(S) origin, so providers sharing an origin share the configured budget.
FAQ
What is the main job of upstream worker queue limits?
LLM Proxy combines server.workers and server.queue_size concurrency bounds with server.upstream_rate_limits rolling-window rules applied at actual upstream admission.
Who should read this reliability resource?
Operators who need predictable capacity limits for provider HTTP calls.
Does this page claim provider performance or pricing advantages?
No. The supported claim is about LLM Proxy's documented routing, configuration, management, security, usage, and deployment contracts. Provider cost, speed, rankings, and benchmark claims are not made here.
Where should setup details come from?
Use the main README and implementation notes for current command, config, and endpoint details. This page summarizes the workflow without replacing LLM Proxy documentation.
What should I read next?
A closely related resource is OpenAI background response polling without client loops, which covers OpenAI background response polling.
Related resources
Use this pattern in LLM Proxy
Start from the canonical API reference, then use the management surface when the workflow needs tenant or provider configuration.
Open API reference