Route Requests Across GPT-5.6, Claude, and Gemini with a Unified LLM Gateway

What you'll build

A local LiteLLM proxy that exposes a single OpenAI-compatible /v1/chat/completions endpoint. Behind that one endpoint, it load-balances requests across OpenAI, Anthropic, and Gemini, automatically retries on a different provider if one fails, and reports per-request cost. Your app code never has to know which provider actually served the request.

One naming note up front: this tutorial's title uses "GPT-5.6" as shorthand for "whichever GPT model is current when you read this." No such model has shipped. The working config below uses gpt-4o, a current OpenAI production model at the time of writing, so every command here actually runs. LiteLLM doesn't validate model strings itself, it just passes whatever you put in model: straight to the provider, so swapping in a newer OpenAI model later means changing one line and nothing else.

Prerequisites

• Python 3.9 or newer (python3 --version)
• API keys for the providers you want behind the gateway: an OpenAI key, an Anthropic key, and a Google AI Studio (Gemini) key
• macOS, Linux, or WSL2 on Windows. No compiled dependencies here, so Apple Silicon vs Intel doesn't matter
• Basic comfort with YAML and curl

Step 1: Install LiteLLM

Use a virtual environment so this doesn't collide with other Python projects.

python3 -m venv venv
source venv/bin/activate
pip install 'litellm[proxy]'

Confirm it installed:

pip show litellm

You should see a Version: line. That's enough to confirm the install worked, don't rely on a --version flag on the litellm CLI itself, since not every build supports it consistently.

If you'd rather skip the local Python environment entirely, the official Docker image works the same way:

docker pull ghcr.io/berriai/litellm:main-latest

Step 2: Set your provider credentials

LiteLLM reads provider keys from environment variables. Export them in the shell where you'll run the proxy:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="AIza..."
export LITELLM_MASTER_KEY="sk-gateway-1234"

LITELLM_MASTER_KEY isn't a provider key, it's the credential clients will use to authenticate to your gateway. Pick a random string, don't reuse a provider key here.

Don't put any of these in a committed file. If you want persistence across shells, put the exports in a .env file that's in your .gitignore, or use your OS keychain.

Step 3: Write the gateway config

Create config.yaml:

model_list:
  - model_name: smart-router
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: smart-router
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: smart-router
    litellm_params:
      model: gemini/gemini-2.0-flash
      api_key: os.environ/GEMINI_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  allowed_fails: 2
  cooldown_time: 30

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

A few things worth understanding here, not just copying:

• Giving all three deployments the same model_name (smart-router) is what makes them one logical group. Clients call smart-router, LiteLLM picks a deployment from the group and load-balances across them.
• routing_strategy: simple-shuffle is the default and requires no external infrastructure. It picks randomly (weighted if you set weight per deployment) among healthy deployments. Don't reach for latency-based-routing yet, it needs a Redis backend to store rolling latency stats across requests, more on that in Next Steps.
• num_retries and allowed_fails are your failover knobs. If the OpenAI deployment throws an error, LiteLLM retries against Claude or Gemini in the same group instead of just failing the request. cooldown_time temporarily benches a deployment that's been erroring, so you're not hammering a provider that's down.
• drop_params: true strips provider-unsupported request params instead of erroring, useful since not every provider accepts every OpenAI parameter (like logprobs on some Gemini models).

If you'd rather use a smaller, cheaper OpenAI model for this exercise, swap openai/gpt-4o for openai/gpt-4o-mini, both are current and work identically here.

Step 4: Start the proxy

litellm --config config.yaml --port 4000

You should see log lines confirming each deployment loaded, plus a line saying the proxy is running on http://0.0.0.0:4000. Leave this running in its own terminal tab.

Docker equivalent, mounting your config in:

docker run -p 4000:4000 \
  -e OPENAI_API_KEY -e ANTHROPIC_API_KEY -e GEMINI_API_KEY -e LITELLM_MASTER_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000

Step 5: Call it like any OpenAI endpoint

Raw curl:

curl -i http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-gateway-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "smart-router",
    "messages": [{"role": "user", "content": "Say hello in exactly 5 words."}]
  }'

Or from Python, using the official openai SDK unchanged except for base_url:

from openai import OpenAI

client = OpenAI(
    api_key="sk-gateway-1234",
    base_url="http://localhost:4000/v1",
)

resp = client.chat.completions.create(
    model="smart-router",
    messages=[{"role": "user", "content": "Explain load balancing in one sentence."}],
)
print(resp.choices[0].message.content)

Note the /v1 suffix on base_url. LiteLLM happens to also expose some routes at the root, but the OpenAI SDK (and most other OpenAI-compatible clients) assumes the standard /v1 prefix and will build request paths accordingly. Pointing at root works by accident on LiteLLM specifically; pointing at /v1 works everywhere, so just do that.

Run this a handful of times and you'll notice the model actually answering isn't always the same provider (check the model field in the JSON response to confirm, LiteLLM rewrites it to the real underlying model it hit).

Step 6: Check cost per request

Run the curl command with -i (as above) and look at the response headers. LiteLLM adds x-litellm-response-cost, the dollar cost of that specific call, computed from the provider's published token pricing:

curl -i http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-gateway-1234" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "hi"}]}' \
  | grep -i x-litellm

For latency, time curl ... is enough to eyeball round-trip time locally. For real dashboards across many requests, you want Prometheus, covered in Next Steps.

Verify it works

• The proxy starts without errors and logs all three deployments under smart-router.
• A curl request returns a normal OpenAI-shaped chat completion (choices[0].message.content).
• The x-litellm-response-cost header is present and non-zero.
• Temporarily set one provider's env var to an invalid value (e.g. export OPENAI_API_KEY=invalid), restart the proxy, and send several requests. Some should still succeed, served by Claude or Gemini, proving the failover path actually works instead of just existing on paper.

Troubleshooting

• AuthenticationError: No models match model_name or similar: the model field in your request body has to exactly match a model_name in config.yaml. Typos here are the most common failure.
• 401 from the gateway itself: you forgot Bearer in the Authorization header, or the token doesn't match LITELLM_MASTER_KEY. Note the env var is only read at proxy startup, restart after changing it.
• 401/403 bubbling up from a specific provider: usually a stale or mistyped key. Double-check the exact env var name LiteLLM expects per provider (GEMINI_API_KEY, not GOOGLE_API_KEY, for the gemini/ prefix) and confirm you exported it in the same shell you launched the proxy from.
• All requests failing after one provider goes down: check cooldown_time and allowed_fails, if they're too aggressive the whole group can get benched. Also confirm you actually have more than one deployment under the same model_name, load balancing across a group of one does nothing.

Next steps

• Latency-based routing: switch routing_strategy to latency-based-routing once you're ready, but it needs a Redis instance to store rolling per-deployment latency across requests and processes (a single in-memory process can't do this reliably at any real traffic level). Add redis_host, redis_port, and redis_password under router_settings and point them at a Redis container or managed instance.
• Persistent cost/spend tracking and the built-in UI: point LiteLLM at a Postgres database via DATABASE_URL to unlock the /ui admin dashboard, virtual API keys, and per-team budgets, instead of reading cost off response headers one call at a time.
• Prometheus metrics: add prometheus to litellm_settings.success_callback to get a /metrics endpoint with latency histograms and request counts per model, then wire it into Grafana.
• Production deployment: run this behind a real reverse proxy with TLS, and read LiteLLM's docs on rate limiting and budget alerts before you put real traffic behind it.