LLM Gateways & Routing (LiteLLM & Portkey)

1. Concept Overview & Systemic Problem

When a company develops dozens of different AI-based services without a single management center, infrastructure chaos ensues:

• Each repository has its own API key, making it impossible to identify which team is consuming the most resources.
• When Anthropic or OpenAI experiences a technical failure or exceeds rate limits (429), all internal agents and services suddenly crash.
• The same typical question is posed to the model 50 times a day, wasting money due to the lack of a shared cache.

LLM Gateways (LiteLLM & Portkey) serve as a single intelligent traffic switch between your company's applications and hundreds of cloud and local models.

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 CENTRALIZED LLM GATEWAY                     │
├─────────────────────────────────────────────────────────────┤
│ 1. INCOMING APPLICATION CALLS (All use standard OpenAI API) │
│    • Cursor IDE, Internal Bots, Customer Facing Apps        │
├─────────────────────────────────────────────────────────────┤
│                          │                                  │
│                          ▼ LITELLM / PORTKEY GATEWAY CORE   │
│    ┌─────────────────────────────────────────────────────┐  │
│    │ • Semantic Redis Cache (Check for instant hit)      │  │
│    │ • Spend Tracker (User/Team budget quotas)           │  │
│    │ • Health Checker & Automatic Fallback Engine:       │  │
│    │   Try: Claude 3.7 ➔ If 429/500 ➔ Fallback: DeepSeek-R1 │
│    └─────────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│ 2. DOWNSTREAM PROVIDERS:                                    │
│    • Anthropic | OpenAI | Bedrock | Self-hosted vLLM on VPS │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

01. Configuring Fault-Tolerant Routing in LiteLLM

Automatic failover configuration in config.yaml:

model_list:
  - model_name: production-coder
    litellm_params:
      model: anthropic/claude-3-7-sonnet
      api_key: os.environ/ANTHROPIC_KEY
  - model_name: production-coder
    litellm_params:
      model: openrouter/deepseek/deepseek-r1
      api_key: os.environ/OPENROUTER_KEY

router_settings:
  routing_strategy: "latency-based-routing"
  fallbacks: [{"production-coder": ["openrouter/deepseek/deepseek-r1"]}]

If Claude exceeds latency or returns an error, the request seamlessly switches to DeepSeek.

02. Single Virtual Key for Employees

The gateway generates a unique virtual token for each developer with a $50 monthly balance. The developer connects this key in Cursor or Cline, while the company centrally monitors analytics and controls spending.

4. Production Engineering Scenarios

01. Gateway as a Single Point of Failure

If the LiteLLM server itself goes down, all company services will halt. Deploy the gateway in at least two replicas behind a load balancer (Caddy/Traefik).

02. Blind Semantic Caching

Semantic caching is beneficial for stable facts but dangerous for code: two similar questions with minor variable name differences should not return the same cached code. Set the similarity threshold no lower than 0.96.

03. Monitoring and Analytics

Implement comprehensive logging and monitoring for all requests and responses through the gateway to identify bottlenecks and optimize performance.

5. Pitfalls, Common Mistakes & Security

LLM Gateways are a mandatory layer of mature architecture. Decoupling application logic from specific model vendors ensures continuous service availability, budget control, and the freedom to switch to the best models on the market in seconds.