LLM Failover & Rate Limiting

AI gateway routes requests to tens of LLM models. A model backend starts responding slowly - not failing, just 10x latency. What happens to your system, and what mechanisms should prevent cascade?

Failure

• requests pile up eventually overwhelm the backend CPU / MEM resources goes down
• if queue - backpressure starts building up
  • users experience outage
  • after down controller (flux) tries to bring up the backend
  • backend is bombarded with incomming requests
• If no queue outage happen

Prevent

• introduce a retry + exponantial backoff + jitter + circut breakers
• retry budget in gateway api
• if possible shard the down server
• shed non-essential load if system is overwhelmed

Examples

Retry - bombard with retries

async def with_retry(request, tries=3):
    i = 0
    while i < tries:
        try:
            return await request()
        except Exception as e:
            i += 1

Backoff Retry - wait between retries

async def with_retry(request, tries=3):
    ...
            await asyncio.sleep(3) # <-- wait 3 seconds and retry
            i += 1

Exponential Backoff Retry - the more you fail the more you wait

async def with_retry(request, tries=3):
    ...
            await asyncio.sleep(2**i) # <-- wait 1, 2, 4, 8, ... seconds and retry
            i += 1

Exponential Backoff Retry + Jitter - if 100 users are retrying at the same time - add randomness to prevent spikes when everyone retries in same intervals

async def with_retry(request, tries=3):
    ...
            await asyncio.sleep(2**i + random.randint(1, 5)) # <-- wait 3, 4, 10, 12, ... seconds and retry
            i += 1

Exponential Backoff Retry + jitter + CIRCUT BREAKER - you don't just exhaust your tries and fail - you add a circut breaker

    self.circuit_state = CIRCUT_STATE.CLOSED
    self.circuit_wait_s = 10
    self.circuit_opened_at = None
    self.failure_count = 0
    self.failure_threshold = 5
    self.lock = asyncio.Lock()


    async def with_retry(self, request, tries=3):
        i = 0

        async with self.lock: # <--- protect state against concurrency

            # 2. if circuit open AND circuit still in cooldown - raise
            if self.circuit_state == CIRCUT_STATE.OPEN:
                if (time.perf_counter() - self.circuit_opened_at) < self.circuit_wait_s:
                    raise Exception("circuit")
                else:
                    # 3. cooldown passed - allow call
                    self.circuit_state = CIRCUT_STATE.HALF_OPEN

            elif self.circuit_state == CIRCUT_STATE.HALF_OPEN:
                raise Exception("circuit is HALF_OPEN - waiting...")


        while i < tries:
            try:
                response = await request()
                
                async with self.lock:  # <--- protect state against concurrency

                    # 5. SUCCESS - reset circuit
                    if self.circuit_state == CIRCUT_STATE.HALF_OPEN: 
                        self.circuit_state = CIRCUT_STATE.CLOSED
                        self.failure_count = 0

                return response 

            except Exception as e:
                i += 1

                if i == tries:

                    async with self.lock:  # <--- protect state against concurrency

                        # 1. THIS request failed - check for others
                        if self.circuit_state == CIRCUT_STATE.CLOSED: 
                            self.failure_count += 1
                            
                            # 1. OTHER requests failed too - trip
                            if self.failure_count >= self.failure_threshold:
                                self.trip_circuit()

                        # 4. circuit waited and retried - still fails
                        elif self.circuit_state == CIRCUT_STATE.HALF_OPEN: 
                            self.trip_circuit()
                    
                    raise e

                await asyncio.sleep(2**i + random.randint(1, 5))

    def trip_circuit(self):
        self.circuit_state = CIRCUT_STATE.OPEN # OPEN circuit
        self.circuit_opened_at = time.perf_counter() # remember WHEN
        self.failure_count = 0  # <--- EXPLAINING THIS IS THE DIFFERENCE BETWEEN L4 AND L5 ENGINEER

---

How would you design rate limiting for a multi-tenant LLM gateway - what do you limit on (requests? tokens?), where does state live, and what breaks at scale?

Problem

• rules: limit per hour
• actors: only authorized users with their api keys
• location: in-service(fasted, easiest, doesn't know global limits) / global service(configurable, extra hop, PoF) / gateway (best, not easy)

Solution

• lives in gateway in redis sidecar
• token bucket vs sliding window - bucket we update every interval and substract on user request, on sliding window we have k last seconds where we invalidate older than that uses to refresh limits.
• we limit on tokens because 1 request can be 100k tokens vs 10 requests of 1k tokens.
• how to limit on tokens - we estimate based on history and input size and reserve that amount in limit, after llm responds and we get exact output tokens size - we refund reserved-real. e.g. anthropic has max_tokens

Concerns

• in our case we only have auth users so at large scale our bottleneck is user quota lookup in redis
• if we also need to lookup the user info by ip - that service becomes overwhelmed

Examples

Token Bucket

class RateLimiter:
    def request(self, user_id, tokens_used):
        key = f"{user_id}:bucket"
        
        while True:
            try:
                # WATCH FOR CHANGES
                r.watch(key)
                
                current_tokens_raw = r.hget(key, "tokens")
                current_tokens = int(current_tokens_raw) if current_tokens_raw is not None else 0
                
                if current_tokens < tokens_used:
                    r.unwatch()
                    return False # Rate limited!
                
                # TRANSACTION TO UPDATE
                pipe = r.pipeline(transaction=True)
                pipe.hset(key, "tokens", current_tokens - tokens_used)
                pipe.hset(key, "last_refill", int(time.time()))
                pipe.expire(key, 60 * 60)
                
                # Throws WatchError if another client modified this key
                pipe.execute()
                return True # Tokens successfully deducted!
                
            except redis.WatchError:
                # KEY CHANGED MID-FLIGHT - RETRY
                continue

    def put_bucket(self, key, amount, pipe):
        pipe.multi()
        pipe.hset(key, "tokens", amount) 
        pipe.hset(key, "refilled_at", int(time.time())) 
        pipe.expire(key, 60 * 60) 
        pipe.execute()

    def main(self):
        REFILL_AMOUNT_S = 1

        while True:
            # 1. GET ALL KEYS
            keys = list(r.scan_iter(match='*:bucket'))
            if not keys:
                time.sleep(1)
                continue

            # 2. GET ALL BUCKETS
            pipe = r.pipeline(transaction=False)
            for key in keys: 
                pipe.hgetall(key)
            all_buckets = pipe.execute()

            # 3. UPDATE IN SINGLE NETWORK ROUNDTRIP
            write_pipe = r.pipeline(transaction=False)
            for key, data in zip(keys, all_buckets):
                if not data:
                    continue
                self.put_bucket(key, REFILL_AMOUNT_S, write_pipe)

            result = write_pipe.execute()
            time.sleep(1)

Reserve then Reconcile - LLM budget limiting

import time
import uuid
from typing import Dict, Any, Tuple
import redis

class MockLLM:
    def generate(self, prompt: str, max_tokens: int) -> Dict[str, int]:
        time.sleep(0.1) 
        prompt_tokens = len(prompt.split()) # OR TIKTOKEN
        actual_completion_tokens = int(max_tokens * 0.6) # 60% of MAX
        return {
            "prompt_tokens": prompt_tokens,
            "completion_tokens": actual_completion_tokens,
            "total_tokens": prompt_tokens + actual_completion_tokens
        }

class TokenLimiter:
    def __init__(self, redis_client: redis.Redis, tpm_limit: int = 10000):
        self.redis = redis_client
        self.tpm_limit = tpm_limit
        self.window = 60  # 1-minute window for TPM

        # Lua script ensures atomic check, reservation, and TTL setting
        self._reserve_script = self.redis.register_script("""
            local key = KEYS[1]
            local reserve_amount = tonumber(ARGV[1])
            local limit = tonumber(ARGV[2])
            local window = tonumber(ARGV[3])
            
            local current = tonumber(redis.call('GET', key) or "0")
            if current + reserve_amount > limit then
                return 0  -- Reject: Exceeds limit
            else
                redis.call('INCRBY', key, reserve_amount)
                if current == 0 then
                    redis.call('EXPIRE', key, window)
                end
                return 1  -- Success: Reserved
            end
        """)

    def _get_minute_key(self, user_id: str) -> str:
        current_minute = int(time.time() / 60) # Groups buckets by the current minute timestamp
        return f"ratelimit:{user_id}:{current_minute}"

    def reserve(self, user_id: str, estimated_tokens: int) -> bool:
        key = self._get_minute_key(user_id)
        success = self._reserve_script(keys=[key], args=[estimated_tokens, self.tpm_limit, self.window])
        return bool(success)

    def reconcile(self, user_id: str, estimated_tokens: int, actual_tokens: int):
        key = self._get_minute_key(user_id)
        difference = estimated_tokens - actual_tokens
        if difference > 0:   self.redis.decrby(key, difference) # REFUND
        elif difference < 0: self.redis.incrby(key, abs(difference)) # MODEL WENT OVER ESTIMATE


if __name__ == "__main__":
    r = redis.Redis(host='localhost', port=6379, decode_responses=True)
    limiter = TokenLimiter(redis_client=r, tpm_limit=1000)
    llm = MockLLM()

    user_id = "user_12345"
    prompt = "Write a comprehensive essay about distributed rate limiting patterns."
    max_tokens = 500
    
    # ESTIMATE
    estimated_prompt_tokens = len(prompt.split())
    worst_case_tokens = estimated_prompt_tokens + max_tokens
    print(f"Attempting to reserve {worst_case_tokens} tokens...")
    
    # CHECK
    if limiter.reserve(user_id, worst_case_tokens):
        print("✅ Reservation successful. Calling LLM...")
        # CALL
        response = llm.generate(prompt, max_tokens)
        actual_used = response["total_tokens"]
        print(f"LLM finished. Actual tokens used: {actual_used}")
        
        # REFUND
        limiter.reconcile(user_id, worst_case_tokens, actual_used)
        refunded = worst_case_tokens - actual_used
        print(f"🔄 Reconciled. Refunded {refunded} tokens to user bucket.")
        
    else:
        print("❌ Request dropped: 429 Too Many Requests. Budget exceeded.")