AI Load Balancing

L4 vs L7

AI L7 decodes MCP / A2A to dynamically auth tool calls, token-based rate-limits, prompt injections, guardrails, route based on GPU metrics or LoRA adapters.

Consistent hashing

KV cache should be sticky to GPU instance with message history. Using consistent hash ring instead of round-robin - map Conversation ID <--> GPU Instance. Otherwise Time-to-First-Token delays.

Health checks

GPU VRAM 100%? Request queue depth too high? KV cache is 100% full? - Degraded & Excluded from candidates. Intercept Agent calls if MCP server is down.

Connection draining

Gateway isolates the node from new agent tasks, keeps connection alive for active agents to complete reasoning loops, receive their full text streams, and gracefully persist their session state.

Multiplexed connections

sending independent agent streams, LLM prompts, and MCP tool requests simultaneously over a single, long-lived TCP connection, instead of opening a new slow connection for every single message or tool call, the gateway packs them together into one pipe.

MCP: Statefull vs Stateless

Stateless MCP - every tool call isolated event, no memory of interactions, trivial to scale behind a standard L4 or L7 balancer. Stateful MCP - maintain session history across turns, meaning use sticky sessions or consistent hashing to route requests same server. Future - Stateless and self-contained.