LiteLLM proxy tracker: load balancing and fallback behavior

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

In complex AI architectures, inference is no longer tied to a single API key with a single provider. Anyone running multiple agents simultaneously or supporting production workflows will inevitably run into rate limits (HTTP 429), upstream timeouts (HTTP 504), and sudden provider outages. LiteLLM Proxy has established itself as a widely used open-source API gateway for putting a uniform OpenAI-compatible interface over dozens of model providers.

In practice, however, configuring a proxy turns out to sound simpler than mastering the dynamic runtime behavior. Load balancing across heterogeneous backends, correctly passing through context limits, and catching stalling streaming responses require in-depth knowledge of the underlying router. In this tracker, we analyze how load balancing and fallback cascades conceptually behave under heavy load, where the structural weaknesses lie, and how parameters should be tuned in production setups to prevent data loss and unnecessary delays.

LiteLLM's routing strategies under the hood

LiteLLM Proxy offers several built-in routing algorithms via the parameter routing_strategy. The choice of this strategy fundamentally determines how incoming requests are distributed across the configured endpoints within a model group. The simplest implementation is simple-shuffle, which amounts to a random selection without taking previous load or response times into account.

For production environments where peak loads and changing provider conditions occur, three more advanced approaches are available:

• least-busy: Sends the request to the endpoint that currently has the fewest active open connections. This prevents a slow endpoint from being flooded with pending connections, but requires precise local state tracking.
• latency-based-routing: Dynamically selects the endpoint with the lowest average response time over a sliding window. This is suitable when the same models are hosted across different regions or cloud providers with varying network latency.
• usage-based-routing-v2: Tracks in a central Redis layer how many tokens per minute (TPM) and requests per minute (RPM) have been sent to each endpoint, and balances requests so that rate limits at upstream providers are proportionally avoided.

To understand how this routing translates into cost control at large-scale operations, consulting the ongoing price list per million tokens offers a clear reference framework for the financial impact of provider choices. After all, a balancer without contextual limits can unnoticeably route traffic to a significantly more expensive fallback model.

Configuration: RPM, TPM, and weighted endpoints

When multiple API keys from the same provider or different deployments (such as Azure OpenAI alongside OpenAI Direct) are combined, the endpoints must be explicitly given capacity limits. LiteLLM uses this metadata to proactively throttle or divert requests before the upstream provider generates an HTTP 429.

Below is a typical configuration in which a model group is spread across Azure, OpenAI, and a locally hosted vLLM cluster:

model_list:
  - model_name: gpt-4o-productie
    litellm_params:
      model: azure/gpt-4o-eastus
      api_base: https://instance-east.openai.azure.com/
      api_key: os.environ/AZURE_EAST_KEY
      rpm: 2400
      tpm: 180000
  - model_name: gpt-4o-productie
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_PROD_KEY
      rpm: 5000
      tpm: 450000
  - model_name: gpt-4o-productie
    litellm_params:
      model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
      api_base: http://vllm-cluster.internal:8000/v1
      api_key: none
      rpm: 1200
      tpm: 90000

router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: redis-cluster.internal
  redis_port: 6379
  redis_password: os.environ/REDIS_AUTH
  enable_pre_call_checks: true

The parameter enable_pre_call_checks: true forces the router to check in Redis before sending whether the estimated token consumption fits within the remaining TPM window of the selected endpoint. This significantly reduces the risk of overloading individual API keys.

For developers looking for optimizations in prompt overhead and payload reduction, the techniques from the article on proven token savings from the community connect directly to effectively using these tight TPM budgets.

Fallback cascades: order, error codes, and cooldown mechanisms

A robust fallback architecture doesn't just absorb capacity issues but also responds to infrastructural and functional failures such as network outages and model unavailability. LiteLLM uses a two-stage approach: retries on the same model (if there are multiple endpoints) followed by a fallback to an alternative model group.

The fallback order is defined in the configuration via fallbacks. A crucial aspect here is the cooldown time: as soon as an endpoint fails with a 5xx or 429 error, LiteLLM marks this endpoint as inactive for a configurable period via cooldown_time.

When an error occurs that does trigger a fallback, the proxy runs through the configured list. If the primary tier (for example Claude 3.5 Sonnet) is unreachable, the router switches to an alternative endpoint such as GPT-4o, and as a last resort to a local open-weight model.

Anyone wanting to compare their own architecture with general routing patterns can consult the guide on orchestrating multiple models via routing and fallback to get a clear picture of the differences between gateway-level routing and application-level routing.

The architecture of state management: Redis vs. in-memory

LiteLLM Proxy can run in a stateless mode where each worker keeps its own statistics in-memory, or in a distributed mode with Redis as a central state store. In production with multiple workers or clustered containers, a central Redis instance is necessary to make consistent decisions.

Without Redis, the split-brain problem occurs: instance A doesn't know that instance B just sent a large prompt batch to the same endpoint. Both instances assume they're operating within their TPM limit, so they forward requests simultaneously and still run into upstream rate limits. Redis synchronizes three essential tables:

1. Token buckets: For tracking current RPM and TPM usage per model alias and per API key.
2. Cooldown registers: A shared status of endpoints that have been temporarily taken out of service due to error messages.
3. Latency matrices: Average response times for latency-based routing.

In addition, caching plays a major role in relieving the load on the routing layer. By enabling semantic or exact response caching directly in Redis, identical prompts don't need to be resent to the providers. For advanced scenarios in which multi-agent systems perform repeating queries, the article on caching architectures for multi-agent loops offers in-depth strategies for using Redis optimally for this.

Streaming responses and the fallback dilemma

Combining Server-Sent Events (SSE) streaming with automatic fallbacks is one of the most complex challenges in LLM infrastructure. The problem arises as soon as the proxy has already forwarded the HTTP 200 headers and the first streaming chunks (such as the role and the first tokens) to the client.

If the upstream provider crashes halfway through generation, the connection drops, or it produces a timeout, LiteLLM can't simply restart the connection on a fallback model without the client receiving corrupted data. After all, the client has already received a partial data stream.

There are two ways to deal with this limitation:

• Direct streaming without buffering: Maximum speed and minimal delay to the first token (TTFT). If the endpoint fails halfway through the stream, the connection drops with an error message. In that case, the client application must itself detect that the stream was incomplete and initiate a retry.
• Buffering the initial chunks: The proxy buffers the first tokens until the connection proves stable before sending the headers to the client. This introduces a slight delay in TTFT, but allows the gateway to still transparently switch to a fallback model in case of an immediate failure at the start.

For a detailed look at how client applications should handle interrupted streams and neatly closing off half-generated JSON, the dossier on streaming with fallback and partial answers offers concrete implementation patterns.

Observability and metrics: what should you monitor?

A routing layer without observability is a blind system. LiteLLM offers native integrations with OpenTelemetry, Prometheus, Langfuse, and Datadog. To determine whether the chosen load balancing strategy is functioning effectively, specific core metrics must be monitored.

# Prometheus metrieken configuratie in litellm config.yaml
general_settings:
  telemetry: false

litellm_settings:
  callbacks: ["prometheus", "otel"]
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]

The most important indicators on the dashboard are:

• litellm_deployment_latency_per_output_token: Provides insight into the actual inference speed per backend, corrected for prompt length.
• litellm_deployment_failed_requests_total: Should be broken down per model and per status code to flag patterns in 429 or 5xx errors early.
• litellm_deployment_cooled_down: Shows in real time which endpoints have been temporarily taken out of service by the cooldown mechanism.
• litellm_deployment_successful_fallbacks: Measures how often a request only succeeded on a secondary or tertiary endpoint. A sustained increase indicates structural undercapacity of the primary tier.

An overview of the broader landscape of monitoring tools and tracing frameworks can be found in the analysis of AI observability and tooling signals.

Analysis of failure modes in a heterogeneous setup

To understand how routing fails under load, it helps to analyze a conceptual setup in which three types of backends work together: a commercial cloud endpoint with tight rate limits (Tier 1), a secondary cloud provider with more generous limits (Tier 2), and a locally hosted model server such as vLLM.

When a simple simple-shuffle strategy is used under peak load, requests are distributed blindly. As soon as the cloud endpoint reaches its TPM limit, chain reactions occur: retries load the proxy further and increase the wait time for all subsequent requests. With a dynamic strategy such as usage-based-routing-v2 with pre-call checks, the router calculates the remaining capacity in advance and diverts requests to the remaining tiers in time.

An important consideration with local fallbacks is 'cold start' latency: when a local model server suddenly receives a wave of requests diverted from the cloud, the processing time per token can temporarily increase because the server has to reallocate its memory buffers and dynamic KV cache. This underscores that a fallback target must also be operationally prepared for abrupt shifts in traffic volume.

Checklist for production implementations

LiteLLM Proxy forms a powerful link in multi-provider AI architectures, provided the routing and fallback mechanisms are tuned to the operational reality of the underlying providers.

The most important design rules for a stable implementation:

• Connect a central Redis instance for distributed token tracking and cooldown registration as soon as multiple proxy instances are running.
• Set explicit rpm and tpm limits on each individual endpoint, preferably with a safety margin below the theoretical provider quota.
• Choose usage-based-routing-v2 for heterogeneous cloud accounts and least-busy for locally hosted model servers with variable compute times.
• Define a strict request_timeout on primary endpoints to prevent stalling connections from blocking the queue.
• Keep in mind that streaming responses can't switch over transparently after the initial tokens in case of provider crashes without application intervention.