LLM Gateway Cache/CAP

Part 1: Caching Patterns

• Cache-aside: The application checks the cache; if it misses, it fetches from the database and loads it into the cache.

• Traditional: Checking Redis for a user profile.

• LLM Gateway: Semantic Caching. The gateway embeds the user's prompt and checks a vector cache for >95% similarity. If missed, it calls OpenAI, then caches the generated response.

• Write-through: The application writes to the cache, which synchronously writes to the database.

• Traditional: Updating a password.

• LLM Gateway: Updating agent configuration states where immediate consistency is required before the next tool call is made.

• Invalidation & TTLs (Time-To-Live): Purging stale data based on events (invalidation) or time limits (TTLs).

• Traditional: Evicting an article from cache when it's edited.

• LLM Gateway: RAG (Retrieval-Augmented Generation) document caches need short TTLs. If grounding documents change but the cache doesn't, the LLM hallucinates outdated facts.

• Hot-key Mitigation: Preventing a single cache node from crashing because everyone is requesting the exact same key.

• Traditional: Adding random jitter to TTLs or using local in-memory caching for a viral tweet.

• LLM Gateway: Mitigating massive spikes when a default system prompt or a viral template is fetched thousands of times per second by edge nodes.

• CDN Layer: Caching assets geographically close to users.

• Traditional: Serving images and CSS.

• LLM Gateway: Great for the chatbot UI, but terrible for LLM generation. You cannot CDN-cache streamed token generation (text/event-stream), as every request is highly dynamic.

Where caching helps/hurts an LLM Gateway:

• Helps: Massive cost savings on LLM provider tokens and massive latency reduction (from seconds to milliseconds) via semantic caching.
• Hurts: "Fuzzy" semantic matches can accidentally return an answer meant for a slightly different question. High memory costs for storing vector embeddings.

---

Part 2: Data Replication & Consistency (DDIA Ch. 5-9)

• Replication (Leader/Follower): One database node handles writes (Leader), while copies handle reads (Followers) to scale traffic.

• Traditional: Storing order logs.

• LLM Gateway: Chat History. Gateways write a user's prompt to the Leader, but pull massive historical chat contexts from Followers to pack into the context window for the next turn.

• Partitioning (Sharding): Splitting a massive database into smaller chunks based on a key (e.g., A-M goes to Node 1, N-Z to Node 2).

• Traditional: Sharding users by geographic region.

• LLM Gateway: Multi-tenant Vector Databases. You must partition vector indices by tenant_id or customer_id. If you don't, an LLM might retrieve Company A's private documents to answer Company B's question.

• CAP Theorem in Practice: You must choose between Consistency (everyone sees the same data) and Availability (the system stays up) during a network Partition.

• Traditional: Bank balances require Consistency (fail the request if nodes can't sync).

• LLM Gateway: Availability wins. If an agent gateway's RAG database is slightly out of sync, it is better to generate an answer with 5-minute-old context than to crash and return a 500 Error to the user.

• Read-your-writes: A consistency guarantee ensuring that if a user updates data, they see that update immediately on reload.

• Traditional: Updating your bio and seeing it instantly.

• LLM Gateway: Crucial for conversational UX. If a user submits a prompt, it must immediately appear in their chat history. If read replication lag causes their own message to vanish on a page refresh, trust in the AI plummets.

• Transactions vs. Sagas: Transactions are rigid, all-or-nothing database updates (ACID). Sagas are multi-step workflows where, if step 3 fails, the system executes compensation logic to undo steps 1 and 2.

• Traditional: A transaction deducts funds and ships an item simultaneously.

• LLM Gateway: Agentic Workflows are Sagas. An AI Agent cannot use a strict database transaction to book a flight (Tool 1), book a hotel (Tool 2), and rent a car (Tool 3) via external APIs. If the car rental fails, the Agent must autonomously trigger a "cancel_booking" tool for the flight and hotel to simulate a rollback.