Validation
Live provider smoke tests for LLM Proxy
Run paid-provider smoke checks separately from CI when real upstream behavior needs verification.
Operators validating provider credentials and hosted provider routes after config or deployment changes.
The problem
CI should be deterministic and avoid paid provider calls, but some changes still need live confirmation against real upstream providers.
How LLM Proxy helps
LLM Proxy keeps live provider smoke tests outside make ci, parses provider keys from a dotenv file without executing shell code, and forces the temporary proxy into management-disabled mode.
How it works
- Put provider API keys in an ignored env file.
- Run make test-live-providers LIVE_ENV_FILE=configs/.env.
- Limit the provider set with LLM_PROXY_LIVE_PROVIDERS when needed.
- Use --write-config to inspect the isolated temporary contract without making a provider call.
- Use model overrides only when debugging a specific provider/model pair.
Feature-to-benefit table
| Feature | Why it matters here | Example |
|---|---|---|
| Separate from CI | Live provider smoke tests are not part of make ci. | Routine validation stays deterministic. |
| Provider discovery and isolation | The dynamic target runs providers whose keys are available through a management-disabled temporary config. | Missing optional keys and hosted management state do not affect unrelated smoke checks. |
| Default model path | By default, smoke requests omit model so provider configured defaults are tested. | The test exercises the default-selection contract. |
Use-case examples
OpenAI smoke
Run LLM_PROXY_LIVE_PROVIDERS=openai with a live env file.
Gemini wrapper
Use make test-live-gemini as the Gemini-specific compatibility target.
Post-deploy check
After publishing, run selected live smokes for providers touched by config changes.
Objections and limitations
- Live smoke tests can call paid APIs.
- They depend on secret availability and provider uptime.
- They are complementary to, not replacements for, black-box CI tests.
FAQ
What is the main job of live provider smoke tests?
LLM Proxy keeps live provider smoke tests outside make ci, parses provider keys from a dotenv file without executing shell code, and forces the temporary proxy into management-disabled mode.
Who should read this validation resource?
Operators validating provider credentials and hosted provider routes after config or deployment changes.
Does this page claim provider performance or pricing advantages?
No. The supported claim is about LLM Proxy's documented routing, configuration, management, security, usage, and deployment contracts. Provider cost, speed, rankings, and benchmark claims are not made here.
Where should setup details come from?
Use the main README and implementation notes for current command, config, and endpoint details. This page summarizes the workflow without replacing LLM Proxy documentation.
What should I read next?
A closely related resource is Internal AI gateway for durable product integrations, which covers internal AI gateway.
Related resources
Use this pattern in LLM Proxy
Start from the canonical API reference, then use the management surface when the workflow needs tenant or provider configuration.
Open API reference