IntroductionIn a previous article, I described building an agentic search framework in Go. While that architecture handled the functional requirements well, operating it at scale revealed significant cost and latency challenges. At millions of queries per month, LLM API costs, and P95 latency approached 5 seconds.This article presents the semantic caching architecture we implemented to address these issues. The system reduced LLM costs by 45-50% and improved P95 latency to under 2 seconds, while maintaining response freshness guarantees.The key insight: caching at multiple granularities within the agentic pipeline provides better results than end-to-end response caching alone. Specifically, caching the agent's planning decisions—which are deterministic and independent of result freshness—achieved a 50% hit rate even with conservative similarity thresholds.Problem AnalysisCost StructureA single query in an agentic search system involves multiple LLM calls:Planning/Tool Selection (~8,500 input tokens): Agent reads tool definitions and decides which tools to invokeTool Execution (minimal cost): External API callsSummarization (~24,000 tokens): LLM formats tool outputs into natural languageAnd couple of smaller models for rewriting the query, and selecting the tools based on the query, and tool responses.Why Traditional Caching FailsString-based caching provides minimal hit rates for natural language queries:"weather in san francisco" → cache key: hash_1"what's the weather in sf" → cache key: hash_2 (miss)"san francisco weather" → cache key: hash_3 (miss)"tell me about san francisco weather"→ cache key: hash_4 (miss)These queries are semantically identical but produce different cache keys. Our initial implementation with exact string matching achieved only ~15% hit rate.The Freshness ChallengeSemantic similarity matching introduces a new problem: distinguishing between queries that should produce identical responses versus those requiring fresh data.Consider:"who invented the telephone" (answer never changes)"what's the weather today" (answer changes daily)Both queries might have similar embeddings to cached entries, but only the first should retrieve cached results. This requires query classification and validation beyond pure similarity matching.Architecture DesignMulti-Layer Caching StrategyRather than caching only the final output, we implemented caching at three distinct layers in the processing pipeline:Query Input ↓┌─────────────────────────────┐│ Agent Planning & Tool │ ← Layer 1: Planner Cache (50% hit rate)│ Selection │ Cache tool calls, not tool results└─────────────────────────────┘ ↓┌─────────────────────────────┐│ Tool Execution │ No caching (requires fresh data)└─────────────────────────────┘ ↓┌─────────────────────────────┐│ Response Generation/ │ ← Layer 2: Summarization Cache (35% hit rate)│ Summarization │ Cache only when tools used static data└─────────────────────────────┘ ↓Final Response ← Layer 3: End-to-End Cache (3% hit rate)Each layer targets different characteristics:Layer 1 (Planning): High hit rate, deterministic outputs, no freshness concernsLayer 2 (Summarization): Medium hit rate, requires freshness validationLayer 3 (End-to-End): Low hit rate but zero computation when hits occurCore ComponentsThe semantic cache system consists of:Embedding Generator: Converts queries to dense vector representations (768-dimensional)Vector Database (ANN): Stores embeddings with approximate nearest neighbor searchKey-Value Store: Stores response payloads separately from embeddingsQuery Classifier: Categorizes queries as evergreen vs time-sensitiveImplementationLayer 1: Agent Planning CacheThe agent planning step is particularly well-suited for caching because:Tool selection depends only on query intent, not result freshnessPlanning outputs are deterministic for semantically similar queriesThis is the most expensive operation (6,500+ tokens per call)type PlannerCache struct { embeddingClient EmbeddingClient vectorDB VectorDB kvStore KVStore}type CacheContext struct { ModelVersion string ToolVersions []string // sorted Locale string EmbeddingVersion string Temperature float32}func (pc *PlannerCache) Get(query string, ctx CacheContext) (*ToolCalls, bool) { // Generate embedding embedding := pc.embeddingClient.Embed(query) // Create cache key from context contextHash := hashContext(ctx) // Search with conservative threshold candidates := pc.vectorDB.Search(VectorSearchRequest{ Embedding: embedding, Tags: contextHash, Threshold: 0.98, // Near-exact matching Limit: 10, }) // Verify context match and retrieve from KV store for _, candidate := range candidates { if candidate.ContextHash == contextHash { if toolCalls := pc.kvStore.Get(candidate.Key); toolCalls != nil { return toolCalls, true } } } return nil, false}func (pc *PlannerCache) Set(query string, ctx CacheContext, toolCalls *ToolCalls, ttl time.Duration) { embedding := pc.embeddingClient.Embed(query) contextHash := hashContext(ctx) key := generateKey(embedding, contextHash) // Store in both vector DB (for similarity search) and KV store (for retrieval) pc.vectorDB.Insert(embedding, key, contextHash) pc.kvStore.Set(key, toolCalls, ttl)}Cache Context Importance: Including model version, tool versions, and other configuration parameters in the cache key prevents serving stale plans after system updates. When tool definitions change, the context hash changes, effectively invalidating old cache entries.Layer 2: Summarization Cache with Freshness ValidationThe summarization cache requires additional logic to prevent serving stale responses:func shouldCacheSummarization(query Query, toolResults []ToolResult) bool { // Multi-turn queries have context dependencies if query.IsFollowUp { return false } // Check tool result freshness for _, result := range toolResults { switch result.Source { case "web_fresh", "web_daily": // Results from frequently-updated sources return false case "static_index": // Results from weekly-updated index are cacheable continue } } // Additional constraints return query.IsSimpleQuery && query.IsSingleStep && len(toolResults) == 1}This conservative approach accepts a lower hit rate (35%) to ensure response freshness. Only queries that exclusively use static data sources are cached.Layer 3: End-to-End CacheThe end-to-end cache serves as a catch-all for repeated identical queries:func (sc *SemanticCache) GetEndToEnd(query string, ctx CacheContext) (*Response, bool) { // First try exact match exactKey := md5Hash(normalize(query)) if resp := sc.kvStore.Get(exactKey); resp != nil { return resp, true } // Fall back to semantic search with high threshold embedding := sc.embeddingClient.Embed(query) candidates := sc.vectorDB.Search(VectorSearchRequest{ Embedding: embedding, Tags: hashContext(ctx), Threshold: 0.98, Limit: 5, }) for _, candidate := range candidates { if resp := sc.kvStore.Get(candidate.Key); resp != nil { return resp, true } } return nil, false}Freshness Control: Two-Stage GatingTo prevent serving stale responses, we implement a two-stage freshness check:Stage 1: Query ClassificationA BERT-based classifier categorizes queries as evergreen (static answers) or time-sensitive (dynamic answers):type QueryClassifier struct { model BERTClassifier}func (qc *QueryClassifier) Predict(query string) (isEvergreen bool, confidence float64) { features := qc.extractFeatures(query) logits := qc.model.Forward(features) isEvergreen = logits[0] > logits[1] // [evergreen_logit, time_sensitive_logit] confidence = softmax(logits)[0] if isEvergreen else softmax(logits)[1] return isEvergreen, confidence}The classifier is trained with 95%+ precision at the cost of recall. This conservative tuning ensures we rarely cache time-sensitive queries incorrectly.Stage 2: Post-Execution ValidationEven if the classifier isn't confident, we can still cache if we verify that only static data sources were used:func (sc *SemanticCache) GetOrCompute( query string, ctx CacheContext, compute func() (Response, error),) (Response, error) { // Stage 1: Query classification isEvergreen, confidence := sc.classifier.Predict(query) if isEvergreen && confidence > 0.95 { // High confidence evergreen - try cache if cached, found := sc.Get(query, ctx); found { return cached, nil } } // Execute computation response, err := compute() if err != nil { return nil, err } // Stage 2: Validate based on execution results shouldCache := false if isEvergreen && confidence > 0.95 { shouldCache = true } else if response.OnlyUsedStaticSources() { shouldCache = true } if shouldCache && response.IsCacheable() { sc.Set(query, ctx, response, 7*24*time.Hour) } return response, nil}Embedding GenerationWe use a 768-dimensional embedding model (similar to BERT base) for semantic similarity:type EmbeddingClient struct { endpoint string model string}func (ec *EmbeddingClient) Embed(text string) []float32 { // Normalize text normalized := strings.ToLower(strings.TrimSpace(text)) // Call embedding service resp := ec.callEmbeddingAPI(EmbeddingRequest{ Text: normalized, Model: ec.model, }) // L2 normalize for cosine similarity return l2Normalize(resp.Embedding)}func l2Normalize(vec []float32) []float32 { var norm float32 for _, v := range vec { norm += v * v } norm = sqrt(norm) normalized := make([]float32, len(vec)) for i, v := range vec { normalized[i] = v / norm } return normalized}Latency: ~15-20ms per embedding generation call.Vector Search with ANNFor efficient similarity search at scale, we use an approximate nearest neighbor (ANN) index:type VectorDB struct { index HNSWIndex // Hierarchical Navigable Small World graph metadata map[string]Metadata}func NewVectorDB(dimension int) *VectorDB { return &VectorDB{ index: NewHNSWIndex(HNSWConfig{ Dimension: dimension, M: 16, // connections per node EfConstruction: 200, // search quality during construction Metric: "cosine", }), metadata: make(map[string]Metadata), }}func (vdb *VectorDB) Search(req VectorSearchRequest) []Candidate { // ANN search returns approximate nearest neighbors neighbors := vdb.index.Search(req.Embedding, req.Limit*2) var candidates []Candidate for _, neighbor := range neighbors { // Filter by tags and threshold meta := vdb.metadata[neighbor.ID] if meta.Tags == req.Tags && neighbor.Similarity >= req.Threshold { candidates = append(candidates, Candidate{ Key: neighbor.ID, Similarity: neighbor.Similarity, ContextHash: meta.Tags, }) } if len(candidates) >= req.Limit { break } } return candidates}Latency: ~10-15ms for search across millions of vectors.Production ResultsAfter deploying the multi-layer semantic cache to production (serving 10M+ queries/month), we observed:Hit Rates by LayerLayerHit RateAvg Latency SavedPlanner Cache44%380msSummarization Cache35%950msEnd-to-End Cache18%1,850msAggregate ImpactCost Reduction: 48% reductionLatency Improvement: P95 latency from 3.2s to 1.9s (41% reduction)Freshness: Zero incidents of stale responses since implementing two-stage gatingLayer Contribution to SavingsThe planner cache (Layer 1) contributes disproportionately to total savings:Planner cache: ~56% of cost savingsSummarization cache: ~30% of cost savingsEnd-to-end cache: ~10% of cost savingsThis validates the strategy of caching deterministic intermediate computations rather than focusing solely on final outputs.Design Considerations and Trade-offsSimilarity Threshold SelectionWe experimented with thresholds from 0.85 to 0.99:Threshold < 0.90: Unacceptable false positive rate. Example: "weather in Seattle" matched "weather in San Francisco"Threshold 0.90-0.95: Better hit rate but occasional semantic mismatchesThreshold ≥ 0.98: Near-exact matching, very low false positive rateWe chose 0.98 as the default. This conservative approach sacrifices some hit rate for quality guarantees.TTL StrategyCache TTLs vary by layer:Planner cache: 7 days (tool definitions change infrequently)Summarization cache: 2 days (more conservative due to potential freshness issues)End-to-end cache: 1 daysShorter TTLs provide additional freshness guarantees at the cost of reduced hit rates.Cache Context GranularityIncluding too many parameters in cache context reduces hit rate. Including too few causes incorrect cache hits after configuration changes.Our context includes:Model name and versionTool definitions and versions (sorted for consistency)User localeEmbedding model versionTemperature and top-p parametersWe exclude:Request timestampUser IDSession IDEmbedding Model SelectionWe evaluated several embedding models:BERT-base (768-dim): Baseline performanceSentence-BERT (768-dim): Similar performance to BERT-baseMiniLM (384-dim): Faster but slightly lower qualityLarger models (1024-dim+): Marginal improvement, significant latency costWe selected a 768-dimensional model as the best performance/latency trade-off.Operational ConsiderationsMonitoringCritical metrics tracked in production:type CacheMetrics struct { HitRate float64 // by layer MissRate float64 // by layer LatencySaved time.Duration CostSaved float64 FalsePositiveRate float64 // classifier accuracy CacheSize int64 EvictionRate float64}We alert on:Hit rate drops >10% week-over-weekFalse positive rate >5% for evergreen classifierP99 cache lookup latency >100msCache InvalidationBeyond TTL-based expiry, we implement targeted invalidation:func (sc *SemanticCache) InvalidateByContext(ctx CacheContext) { contextHash := hashContext(ctx) // Find all cache entries with this context keys := sc.vectorDB.GetKeysByTag(contextHash) // Delete from both vector DB and KV store for _, key := range keys { sc.vectorDB.Delete(key) sc.kvStore.Delete(key) }}This allows immediate invalidation when tool definitions or models are updated, rather than waiting for TTL expiry.Storage RequirementsApproximate storage per 1M cached queries:Vector DB: ~3GB (768-dim floats + metadata)KV Store: ~5GB (compressed response payloads)Total: ~8GB per 1M cache entriesWith 7-day TTLs and 10M queries/month, steady-state storage is approximately 50-60GB.ConclusionMulti-layer semantic caching is a critical component for production LLM systems operating at scale. By caching at multiple granularities—particularly at the agent planning layer—we achieved significant cost reduction (48%) and latency improvement (41%) while maintaining response quality and freshness.The key architectural insight is that deterministic intermediate computations (agent planning, tool selection) are more cacheable than final outputs, which depend on fresh data. This inverts the typical caching strategy and provides better hit rates where they matter most.For teams building agentic systems, the planner cache should be the first caching layer implemented. It provides the highest return on investment and requires no freshness validation.ReferencesPrevious article: Building a Production-Ready Agentic Search Framework