A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware

AI summary

Overview

Research area: Distributed systems and caching infrastructure for AI-powered web search (cs.DC), specifically multi-tier cache design for stateful LLM answer engines.

Technical level: Intermediate — the paper assumes familiarity with Redis, TTL-based eviction, embedding vectors, and cosine similarity, but explains each design decision in plain terms.

Scope: A single-paper presentation of a three-layer caching architecture (session context, semantic query deduplication, URL embedding reuse) evaluated from a historical production deployment of the OreoLook answer engine on commodity CPU hardware.

What This Paper Is About

Commercial AI search products charge per-query API fees that scale linearly with volume, making them expensive to run at scale. The authors built OreoLook (formerly lixSearch), an open-source answer engine that performs web search with automated browser agents and routes synthesis to a remote LLM provider, keeping all local search, caching, session-management, and embedding work on commodity CPU hardware. As usage grew, three failures emerged — sessions lost context, rephrased queries triggered redundant work, and the same URLs were embedded repeatedly across sessions. The paper describes and evaluates the three-layer cache they built to address each failure.

Key Contributions

1. A unified three-layer cache architecture partitioning a single Redis instance into three logical databases: DB 2 for a rolling session context window, DB 0 for a per-session semantic query cache, and DB 1 for a global cross-session URL embedding cache. The paper states that no prior system it surveys addresses session persistence, semantic deduplication, and embedding reuse together in one lightweight package.

2. A hybrid hot/cold storage design in which the Redis hot window holds the most recent k = 20 messages per session and overflow is serialized into Huffman-compressed .huff disk archives, with transparent re-hydration on demand and a background LRU daemon that migrates idle sessions to disk before freeing Redis memory.

3. A pure-Python canonical Huffman codec with a documented .huff binary format, justified against zlib and lz4 by small-payload efficiency and zero native dependencies rather than by superior compression.

4. A production evaluation snapshot reporting a 89.3% aggregate Redis keyspace hit rate, 0.1 ms Redis read latency, a 1.38 MB total Redis memory footprint, and a measurement-period cost comparison against commercial AI search pricing.

Main Findings

• Aggregate hit rate: Over 6 days and 114,547 total Redis commands, the instance recorded 2,182 keyspace hits and 262 keyspace misses, for a hit rate of 2,182 / (2,182 + 262) = 89.3%. The authors explicitly state this is the Redis-level aggregate across all three databases including internal operations such as TTL refreshes, list reads, and existence checks, and is not a query-level semantic-cache hit rate and is not used to estimate avoided inference cost.

• Memory footprint: The production Redis instance served all three databases with a total footprint of 1.38 MB, of which 96.4% was actual data. At measurement (6 days uptime), DB 0 had 0 keys (all entries expired under the 5-minute TTL, no active queries), DB 1 had 0 keys (24 h TTL expired), and DB 2 had 16 keys with an average TTL of 72,923 s (approximately 20 hours).

• Latency profile (ms, log scale): Redis GET 0.1 ms, pipelined 10-command operation 0.4 ms, Huffman decode 8.0 ms, Huffman encode 8.9 ms, disk read for 6 turns 3.0 ms, disk read for 133 turns 107 ms. The paper notes Redis reads are two orders of magnitude faster than disk, confirming the hot-window design.

• Production compression ratios on five conversation archives, computed as compressed size / raw JSON size excluding the 24-byte application header: c45775-a (133 turns, 73,840 B) 65.3%; dice-4 (7 turns, 4,635 B) 65.2%; shr-26 (10 turns, 4,118 B) 68.8%; ch778-d (6 turns, 2,314 B) 68.8%; ghmd-87 (8 turns, 3,427 B) 72.4%. Average 67.2%.

• Synthetic compression by payload size: 153 B raw → 122 B compressed (79.7%); 765 B → 388 B (50.7%); 1,530 B → 722 B (47.2%); 7,650 B → 3,387 B (44.3%). The paper reports the codec approaches approximately 45% ratio at scale above 1 KB, and that the paper elsewhere describes approximately 54% ratio on synthetic conversation text.

• Codec comparison (lower is better): At 2,314 B, Huffman 68.8% versus zlib-1 63.2% and lz4 72.1%; at 4,118 B, 68.8% versus 54.7% and 65.3%; at 4,635 B, 65.2% versus 53.9% and 64.8%; at 73,840 B, 65.3% versus 38.4% and 47.2%. Zlib achieves better compression at all sizes. The paper states the gap narrows to only 5.6 percentage points at 2 KB and that Huffman consistently outperforms lz4 at all tested sizes.

• Cost comparison (measurement-period snapshot, per query): SearchGPT API $0.03–0.10; Google Gemini Pro $0.035; Google Gemini Flash $0.014; Perplexity Sonar Pro $0.018; Perplexity Sonar $0.005; OreoLook (uncached) $0.015. At 100k queries/month: SearchGPT $3,000–10,000; Gemini Pro $3,500; Gemini Flash $1,400; Sonar Pro $1,800; Sonar $500; OreoLook $1,596.

• OreoLook cost breakdown: A fixed 8-vCPU cloud instance at approximately $96/month, plus variable inference at 0.6 pollen/M input tokens and 3.0 pollen/M output tokens. A typical query consuming approximately 10K input and approximately 2.5K output tokens costs approximately $0.014 in LLM inference, for a combined approximately $0.015 per standard query.

• Initial version cost reduction: Moving from SearchGPT at $0.03–0.10 per query to open-provider LLM inference with browser-agent search brought the per-query cost to approximately $0.02 (only provider inference billed, no per-query search API fees).

• Per-layer contribution (exploratory estimates, not controlled request-level measurements): Layer 1 accounts for an estimated 75–80% of total keyspace hits, with each user message triggering 2–4 Redis reads and an observed average session length of 8 turns across 16 active sessions. Layer 2: approximately 15–20% of queries within a session are semantic near-duplicates at cosine similarity ≥ 0.90, and each semantic hit avoids a full pipeline run, saving 3–8 seconds of wall-clock time. Layer 3: popular URLs appear across 10–30% of sessions, each hit saving approximately 200 ms of embedding computation.

• Embedding storage saving: A 384-dimensional embedding occupies exactly 1,536 bytes in Redis as raw float32 bytes, compared to approximately 3,800 bytes as a JSON array of floats — a 2.5× space saving.

• Single URL embedding cost: Computing a 384-dimensional embedding takes approximately 200 ms per URL.

• Design-decision rationale: The authors report that gzip's dictionary overhead is a 32 KB window, whereas Huffman's symbol table is at most 256 entries and no more than 512 bytes of overhead. They motivate Huffman by byte frequency skew in English conversation text: spaces account for approximately 18% of bytes and the letter 'e' approximately 13%, while 'z' appears only approximately 0.07% of the time.

• Configuration defaults: Rolling window k = 20 messages; semantic similarity threshold τ = 0.90; up to 50 cached pairs per URL; semantic cache TTL 5 minutes; URL embedding cache TTL 24 hours; session context TTL 24 hours; LRU daemon checks every 60 seconds for sessions idle beyond a default 120 minutes; the reported setup mentions a 30-day TTL cleanup expiry in the session lifecycle figure.

• Deployment configuration: A single 8-vCPU Intel Cascade Lake cloud instance (2 GHz, 32 GB RAM, no GPU), three containerized replicas (2 vCPU, 2 GB RAM limit each) with 10 Hypercorn worker processes per replica (30 total), Redis 7.4 in its own container capped at 2 GB, and an nginx load balancer.

Methodology in Plain English

The authors built the answer engine first and let the caching architecture grow out of production pain points rather than designing it upfront. Web search is performed by automated headless browser agents that navigate search engines the way a human would, extracting results and fetching full-page content; that content is sent to provider-routed LLM inference for synthesis. The first version had no RAG, no vector database, and no caching.

Each cache layer was added in response to a specific observed failure. When sessions lost context, they added a rolling window of the k = 20 most recent messages in Redis, with overflow written to Huffman-compressed disk archives and re-hydrated on demand. When users rephrased queries, they added a semantic cache that embeds each incoming query into a 384-dimensional vector with sentence-transformers/all-MiniLM-L6-v2, compares it against cached query embeddings for the same session and URL using cosine similarity, and returns the cached response when the maximum similarity is at least 0.90. When the same URLs were embedded repeatedly, they added a global URL embedding cache storing raw float32 byte arrays shared across all sessions.

All three layers live in one Redis instance but in three separate logical databases (DB 0, DB 1, DB 2) chosen for different TTL profiles, different scopes (per-session versus global), and independent failure modes. A single coordinator object exposes four verbs — add_message_to_context, get_semantic_response, get_url_embedding, and get_stats — so the pipeline integrates through one object per session.

The evaluation is described as a historical production snapshot rather than a controlled experiment. The authors report operational metrics — command counts, keyspace hits and misses, memory footprint, key distributions, per-operation latencies measured in milliseconds, and compression ratios computed from stored archive sizes relative to raw JSON sizes — and derive per-layer contribution estimates from observed key distributions and access patterns. They state explicitly that these per-layer figures are exploratory estimates rather than controlled request-level measurements, and that request-level measurement of semantic-cache hits and avoided provider tokens remains future work.

Why This Matters

Impact on research. The paper adds a documented production data point to a gap it identifies in the literature: existing work covers individual concerns (LangChain's in-process memory buffers, GPTCache's global semantic cache, MemGPT's LLM-driven paging, Semantic Kernel's Redis cache backend) but not all three caching concerns in one lightweight system. Its most methodologically notable move is refusing to convert a Redis keyspace hit rate into a claimed inference-cost saving — the authors state plainly that the 89.3% figure counts internal Redis operations and cannot be converted into the fraction of user queries that bypass inference.

Real-world applications:

• Cost-sensitive answer engines and search assistants that need per-query costs well below the $0.03–0.10 range commercial search APIs charge, by replacing per-query search fees with browser agents and amortized fixed infrastructure.
• Multi-turn conversational assistants requiring session resumption, where users return hours or days later and expect the system to recall prior turns without keeping full histories in memory.
• Retrieval-augmented generation pipelines that touch overlapping URLs, where a global embedding cache with a 24-hour TTL avoids recomputing the same vectors across sessions.
• CPU-only or GPU-constrained deployments, since the authors' local stack — including embedding computation — runs on commodity CPU hardware with synthesis delegated to a remote provider.

Industry relevance. The design targets teams running LLM applications on modest infrastructure where a separate vector database is an operational burden. Storing embeddings directly in Redis as JSON arrays rather than requiring FAISS, Milvus, or Qdrant removes infrastructure the paper argues most caching savings in practice come from the other two layers anyway. The per-layer analysis suggests that session context management, not semantic caching, delivers the majority of cache activity — a finding relevant to anyone deciding where to invest engineering effort first.

Future Directions

• An optional native Huffman extension. The pure-Python codec runs at approximately 800 KB/s, which the authors call acceptable for conversation archives under 10 KB but insufficient for megabyte-scale payloads.
• Pluggable vector storage such as Qdrant for the semantic cache. The current design uses brute-force cosine similarity at O(n) with n ≤ 50, which suffices at present scale but would require a vector index for significantly larger deployments.
• Adaptive TTL policies that learn optimal cache lifetimes from observed query patterns, replacing the fixed 5-minute, 24-hour, and 30-day settings.
• Cross-session knowledge transfer, where insights from one conversation can inform responses in another, plus request-level measurement of semantic-cache hits and avoided provider tokens — which the paper leaves as explicit future work.

Target Audience

Backend and infrastructure engineers building LLM-powered search or conversational products on constrained budgets; distributed-systems practitioners interested in multi-tier cache design with hybrid hot/cold storage; and applied researchers studying cost-efficient retrieval-augmented generation. Readers evaluating caching claims critically will benefit most from the paper's explicit separation between Redis keyspace metrics and query-level cache performance. Readers looking for controlled benchmarking of the semantic cache against GPTCache will not find it here — the authors state they did not benchmark GPTCache directly because the two systems are not drop-in replacements.

Authors’ abstract

AI-powered search products such as ChatGPT search, Google's AI Overviews, and Perplexity provide LLM-synthesized answers grounded in live web results. We developed OreoLook (formerly lixSearch), an open-source answer engine using automated browser agents and provider-routed LLM inference. Its local search, caching, session-management, and embedding stack runs on commodity CPU hardware; answer synthesis is performed by a remote inference provider. As usage grew, sessions lost context, equivalent queries triggered redundant work, and URLs were repeatedly embedded across sessions. We present a three-layer caching architecture: (1) a Session Context Window maintaining a rolling window of recent messages in Redis with automatic overflow to Huffman-compressed disk archives; (2) a Semantic Query Cache catches rephrasings via cosine similarity on embedding vectors, eliminating redundant LLM invocations; and (3) a URL Embedding Cache that deduplicates embedding computations across sessions. Deployed on a single 8-vCPU Intel Cascade Lake server (2 GHz, 32 GB RAM) running 30 Hypercorn worker processes across three containerized replicas, the evaluated system reported an 89.3% aggregate Redis keyspace hit rate with 0.1 ms read latency and just 1.38 MB of memory overhead. A background LRU eviction daemon migrates idle sessions from Redis to disk and re-hydrates them on demand, enabling conversations that can be resumed hours or days later under the configured retention policy.