LLM Proxy

Integrate once

Keep your model options open.

Choose direct HTTP or an official client once, then use these source-backed guides to route across supported providers and models without rebuilding the application boundary.

Choose an integration Explore supported models Open API reference

Integration surfaces

Start with the interface your stack already speaks.

HTTP API Canonical JSON from any language

Use POST /v2 directly with one tenant client key.

 Official Go client Typed messages and media

Construct validated requests and preserve typed failures.

 Official Python client Small synchronous transport

Send canonical messages from scripts and services.

 Official CLI Prompt workflows from the shell

Pipe text from scripts without a provider SDK.

Choose your path

Use the guide closest to the way your team ships.

AI-assisted builders Start from one copyable request

Give prototypes and coding agents a durable API boundary.

 Startups and product teams Compare model fit behind one endpoint

Change provider routing without multiplying product integrations.

 Platform and engineering teams Standardize access across applications

Centralize credentials, supported routes, and usage signals.

Start here

Understand the integrate-once contract.

Multi-provider LLM proxy Integrate once through one multi-provider LLM proxy

See what stays stable, what changes per route, which clients are available, and where the current support boundary lives.

Provider routing

Provider routing resources

multi-provider LLM proxy Integrate once through one multi-provider LLM proxy

Connect through one canonical API or official client, then select any supported text provider and model route without rebuilding the application integration.

 OpenAI Claude Gemini one endpoint Switch OpenAI, Claude, and Gemini behind one endpoint

Send one canonical messages request to supported OpenAI, Anthropic, or Gemini models while LLM Proxy owns each provider's native API contract.

 OpenAI-compatible provider gateway OpenAI-compatible provider gateway

Route Meta Muse Spark, DeepSeek, DashScope Qwen, Kimi, MiniMax, SiliconFlow, Z.AI, and Grok text calls through one compatible adapter.

 Gemini Interactions proxy Gemini Interactions proxy for shared LLM calls

Run model-specific Gemini Interactions lifecycles while callers keep one blocking proxy request.

 Anthropic Claude Messages proxy Anthropic Claude Messages proxy for /v2 callers

Route shared messages to Anthropic's native Messages API without changing client contracts.

Security

Security resources

server-side provider API keys Keep provider API keys server-side for LLM apps

Use tenant secrets for clients while upstream provider credentials stay on the LLM Proxy server.

 tenant secret AI gateway Tenant-secret AI gateway for internal applications

Give client apps one llm-proxy tenant secret while the service owns provider credentials and routing.

 generated LLM proxy secret rotation Rotate generated LLM Proxy client keys with confidence

Automatically create a missing LLM Proxy client key, show it once, store only its digest, and replace it through an explicit confirmed rotation.

 encrypted provider key storage Encrypted provider key storage for managed tenants

Store tenant-owned provider API keys with AES-GCM encryption at rest and honest security wording.

 reject client provider key leaks Reject client-supplied provider key leaks

Fail public proxy requests that try to send upstream provider API keys in query, JSON, or multipart input.

Management UI

Management UI resources

self-service LLM key management Self-service LLM key management for internal teams

Log in to the LLM Proxy app, create a client key for the selected tenant, and autosave one provider API key before leaving Settings.

 bring your own provider key portal Bring-your-own provider key portal for AI access

Use LLM Proxy Settings to let users save provider keys and keep public proxy calls key-free.

 TAuth protected management API TAuth-protected management API for LLM Proxy

Gate account, tenant, provider key, defaults, usage, and admin APIs behind validated TAuth sessions.

 copyable LLM curl examples Copyable LLM curl examples from current profile data

Render copyable LLM curl examples from live profile data while keeping the one-time raw client key out of page markup and copied commands.

 provider-specific system prompts Provider-specific system prompts in LLM Proxy Settings

Store each provider's text model and system prompt with its managed provider configuration.

API contract

API contract resources

v2 chat messages API Canonical /v2 chat messages API for LLM calls

Send ordered system, user, and assistant messages through one v2 endpoint before provider routing.

 large prompt JSON POST Large prompt JSON POST for LLM requests

Use JSON POST bodies when prompts are too large or structured for query-string requests.

 OpenAI web search guardrails OpenAI web search guardrails in an LLM proxy

Expose web_search only when the selected OpenAI model is configured to support it.

 LLM response formats JSON XML CSV text LLM response formats: JSON, XML, CSV, and text

Choose response formatting through format or Accept without changing the provider route.

 per-request provider model selection Per-request provider and model selection

Use provider and model parameters only when a request should override tenant or provider defaults.

 LLM system prompt handling System prompt handling without ambiguous inputs

Keep system instructions explicit across prompt bodies, v2 messages, tenant defaults, and provider settings.

 max tokens provider limit validation max_tokens validation across LLM providers

Map max_tokens to provider-specific fields and reject known invalid token caps before upstream calls.

Dictation

Dictation resources

audio transcription proxy API Audio transcription proxy API behind tenant secrets

Route multipart audio transcription through /dictate using the same tenant-secret boundary as text.

 dictation provider routing Dictation provider routing for OpenAI, Z.AI, Grok, and SiliconFlow

Select dictation-capable providers through /dictate while keeping transcription URLs server-side.

Reliability

Reliability resources

OpenAI background response polling OpenAI background response polling without client loops

Let LLM Proxy own OpenAI Responses background mode and return the final answer in one REST call.

 upstream worker queue limits Upstream worker and queue limits for LLM traffic

Use shared worker and queue controls to bound upstream HTTP operations for text and dictation.

 LLM proxy status code map LLM Proxy status code map for callers

Handle missing keys, bad inputs, rate limits, disabled providers, timeouts, and upstream failures consistently.

 LLM provider overload timeout handling Provider overload and timeout handling for LLM calls

Use clear 503 and 504 behavior for queue pressure and provider work that exceeds the proxy deadline.

Configuration

Configuration resources

LLM model catalog configuration LLM model catalog configuration in config.yml

Keep exact model identity and provider offerings in normalized runtime config.

 provider default model selection Provider default model selection for omitted models

Let omitted model fields resolve through tenant defaults or selected-provider configured defaults.

 strict YAML config placeholders Strict YAML config placeholders for LLM Proxy

Use config.yml as the only service config source and env only for strict placeholder expansion.

 multi-tenant ownership migration Transactional multi-tenant account ownership migration

Upgrade one-tenant-per-user state into explicit TAuth accounts and isolated personal tenants without changing opaque tenant ids.

 GORM managed tenant persistence GORM-managed tenant persistence for LLM Proxy

Persist TAuth accounts, isolated tenants, provider settings, generated secret digests, defaults, and usage through GORM.

Usage

Usage resources

normalized token usage metadata Normalized token usage metadata across providers

Expose request, response, and total token counts through common headers and JSON fields.

 managed tenant usage dashboard Account-wide managed tenant usage dashboard for LLMs

See all owned tenants by default, filter one tenant when needed, and inspect safe failed-request details without exposing secrets.

 admin usage visibility without secrets Admin usage visibility without exposing secrets

Let configured administrators inspect managed users and 30-day usage without raw keys or prompts.

 usage metadata without prompts Usage metadata without storing prompts or responses

Track managed-tenant usage signals while excluding prompt, transcript, audio, response, and secret content.

Deployment

Deployment resources

API-served runtime config static UI API-served runtime config for a static LLM UI

Keep the Pages frontend static while the backend serves current runtime config at /config-ui.yaml.

 local and hosted LLM proxy config Local and hosted LLM Proxy config profiles

Use the same strict config contract locally and in hosted split-origin deployments.

 GitHub Pages LLM management UI GitHub Pages management UI for LLM Proxy

Publish the static management frontend from site/ while the backend owns API and runtime config.

Clients

Clients resources

LLM Proxy client authentication Authenticate an LLM Proxy client with a tenant secret

Configure a client with a tenant secret, send canonical /v2 requests, and keep management sessions and provider keys on separate boundaries.

 Go LLM proxy client v2 Go LLM Proxy client with a v2-only transport

Use the Go package to send canonical messages requests through POST /v2.

 Python LLM proxy client v2 Python LLM Proxy client with v2 messages

Use ClientMessagesRequest and post_messages for canonical text requests from Python.

 installable LLM proxy CLI Installable LLM Proxy CLI for prompt workflows

Use llm-proxy-client to send prompt text as canonical /v2 messages from the command line.

Validation

Validation resources

live provider smoke tests Live provider smoke tests for LLM Proxy

Run paid-provider smoke checks separately from CI when real upstream behavior needs verification.

Use cases

Use cases resources

internal AI gateway Internal AI gateway for durable product integrations

Give institutional teams one AI integration while platform engineering centralizes provider credentials, routing, validation, and usage signals.