You launch an LLM feature.Users like it. Usage grows. Someone adds conversation history. Someone else adds RAG. Then tools. Then retries. Then an evaluator that uses another LLM.A few months later, finance asks a question engineering teams increasingly have to answer:Why is the AI bill growing faster than the product?The first reaction is often to open the system prompt and start deleting adjectives.That is usually the wrong place to begin.Reducing a 1,200-token prompt to 1,050 tokens is useful, but it does not fix an architecture that sends every request to the most expensive model, resends thousands of irrelevant context tokens on every turn, retries silently, generates text nobody displays, and processes overnight jobs through real-time endpoints.LLM cost optimization is increasingly a systems problem.The useful question is not:How do we make this prompt cheaper?It is:Why did this request need this model, this much context, this many calls, and this much output in the first place?That shift changes where you look for savings.Start With the Cost EquationBefore touching the architecture, write down what you are actually paying for.Depending on the provider, a simplified request cost looks something like:request cost = uncached input tokens + cache writes + cache reads + output tokens + tool/API charges + retry cost + supporting infrastructureNot every provider exposes or prices those components identically.That matters.For example, prompt caching can have separate read and write economics. Current OpenAI models can price cached reads substantially below normal input, while explicit cache writes can carry their own pricing. Anthropic similarly separates normal input, cache writes, and cache reads.So the first optimization rule is almost boring:Measure before optimizing.It is also the rule teams skip most often.1. Build Cost Attribution Before Cost OptimizationIf LLM calls are scattered across services, debugging cost becomes surprisingly difficult.A classification call in one service triggers a retrieval pipeline in another, which triggers generation, which fails validation, which retries, which invokes an evaluator.The invoice sees tokens.You need to see why those tokens existed.At minimum, record something like:type LLMUsageEvent = { requestId: string; feature: string; tenantId?: string; model: string; route: "small" | "standard" | "frontier"; promptVersion: string; inputTokens: number; cachedInputTokens?: number; outputTokens: number; purpose: | "user-response" | "classification" | "retrieval" | "summarization" | "evaluation" | "retry" | "background-job"; latencyMs: number; retryCount: number; estimatedCostUsd: number;};The exact schema is less important than the attribution.You want to answer questions such as:Which feature spends the most?Which tenants generate disproportionate usage?Which model handles most requests?How much spend comes from retries?How much comes from internal LLM calls rather than user-visible responses?Which prompt versions increased average context size?What percentage of eligible input is actually being served from cache?What is the cost per successful business outcome?That last question matters most.A support assistant costing $0.08 per conversation may be wonderful economics if it resolves a $6 support interaction.A summarization feature costing $0.03 per click may be terrible economics if almost nobody uses the summary.Cheap tokens are not the goal. Efficient product economics are.2. Route by Difficulty Instead of Sending Everything to Your Best ModelOne of the most powerful architecture changes is also conceptually simple:Not every task deserves your most capable model.A production workload may contain:classification,entity extraction,formatting,summarization,straightforward retrieval synthesis,code generation,ambiguous reasoning,multi-step agentic work,high-risk decisions.Those are not the same workload.They should not automatically have the same inference path.A basic architecture looks like this: ┌── simple ─────> small / fast modelRequest -> Router ───┼── normal ─────> standard model └── difficult ───> frontier model ↑ validation failure ───┘The price spread between model tiers can be meaningful, although the exact ratio changes constantly as providers update their lineups and pricing. The important architectural principle is independent of today's price table: buy only the capability the task requires.Don't Build an ML Router on Day OneYou probably don't need a sophisticated learned router initially.Start with signals you already have:function chooseModel(request: AIRequest): ModelTier { if (request.task === "classification") { return "small"; } if (request.task === "structured-extraction") { return "small"; } if (request.requiresDeepReasoning) { return "frontier"; } if (request.risk === "high") { return "frontier"; } return "standard";}Later, routing can incorporate:historical success rates,request complexity,context length,evaluation scores,model confidence proxies,latency targets,tenant SLA,task-specific classifiers.But routing becomes dangerous when cost is the only optimization target.The correct objective is closer to:minimize costsubject to: quality >= required threshold latency = required thresholdThat is a very different problem from "use the cheapest model."Escalation Makes Routing SaferCheap models should not be dead ends.If a lower-cost model fails schema validation, violates a business rule, produces low-confidence extraction, or fails a task-specific evaluator, escalate the request.Small model | +--- valid ----------> return | +--- validation fail -> Standard model | +--- fail -> Frontier modelYou turn model-routing mistakes into an occasional additional inference rather than a customer-visible failure.The trade-off is latency, so measure escalation rates rather than pretending routing is free.3. Put Your Context on a DietThe next place to look is the request itself.Modern AI applications often accumulate context without anyone intentionally designing a context budget.A request gradually becomes:system instructions+ user profile+ conversation history+ retrieved documents+ tool schemas+ examples+ memory+ intermediate results+ current messageEvery individual addition seemed reasonable.The resulting prompt may not be.The right question is:What information does the model need to solve this request correctly?Not:What information do we happen to have available?Conversation HistorySending the entire conversation forever is easy to implement and expensive to scale.A more deliberate strategy is:stable instructions+ structured conversation state+ summary of older history+ recent verbatim turns+ current requestBut summarization has a quality cost: summaries can discard details.So don't blindly compress everything.Preserve information that must remain exact—identifiers, decisions, constraints, contractual language, code, numerical values—and summarize only history where semantic compression is acceptable.RetrievalRAG systems have their own version of context inflation.A team starts with topK = 10 because ten feels safer than three.Eventually every query carries ten large chunks into generation whether the model needs them or not.There is no universal correct k.k = 3 is not inherently better than k = 10.Instead, treat retrieval as an evaluated budget:retrieval qualityvs.generation qualityvs.tokens addedvs.latencyGood systems may combine:broader candidate retrieval,reranking,deduplication,relevance thresholds,a final context-token budget.Measure answer quality as you reduce context.If you can remove 40% of retrieved text with no measurable degradation, that is evidence.Tool DefinitionsAgentic applications can have another hidden source of context: tools.If the agent knows about dozens of functions, APIs, or MCP tools, you may be repeatedly providing schemas that are irrelevant to the current task.Where your model/API architecture allows it, expose the smallest useful tool set for the current intent.A billing question probably does not need the deployment tools.A calendar request probably does not need your SQL administration schema.Besides cost, this can simplify the model's decision surface.4. Design for Prompt Caching Instead of Merely Enabling ItCaching deserves more nuance than "turn it on."Different providers implement caching differently, but the underlying economic idea is similar:Reusable prompt material should not always require the same computation and price as brand-new context.Current OpenAI pricing, for example, includes substantially discounted cached-input pricing, while Anthropic exposes separate cache-write and cache-read pricing. Google's Gemini API also provides context caching for eligible models and workloads.The architecture implication is important.Separate stable context from volatile context.Conceptually:MOST STABLE│├── system / policy instructions├── stable examples├── stable tool definitions├── relatively stable reference context├── recent conversation├── retrieved data for this request└── current user message│MOST VOLATILEHistorically, prefix reuse has been especially important for cache-friendly request construction, and modern APIs increasingly expose more explicit cache controls as well. OpenAI's current API, for example, supports explicit prompt-caching behavior and extended cache-retention controls in addition to automatic behavior.There is an important catch:A cache write is not necessarily free.Depending on provider and configuration, writing reusable context can cost more than ordinary input, while later cache reads are cheaper. Anthropic explicitly uses this model, and current OpenAI explicit caching also distinguishes write economics.That means caching has a break-even point.A huge static prompt reused hundreds of times may be a fantastic candidate.A giant prompt used twice may not be.Measure:cache hit ratecache read tokenscache write tokenscache lifetimerequests per cached prefixnet dollars savedDon't celebrate "cache enabled."Celebrate lower unit cost.5. Semantic Caching Is Powerful—and Much Easier to Get WrongPrompt caching still executes a model request.A semantic cache tries to avoid the request entirely.Suppose users repeatedly ask:How do I reset my password?I forgot my password.Where can I change my login password?How can I create a new password?For sufficiently stable knowledge, the application can embed the new question, find a semantically equivalent cached query, and reuse a previously generated answer.The architecture looks roughly like:Request |Embed query |Semantic search in response cache | +--- strong safe match ---> cached answer | +--- no match -----------> LLM | +-> cache eligible resultThe expensive part is not implementing nearest-neighbor search.The expensive part is proving that two similar questions can safely share an answer.Cache Identity Must Include More Than TextNever assume:cacheKey = semanticSimilarity(question)is enough.Your effective identity may include:semantic intent+ tenant+ permission scope+ locale+ document/version+ product version+ prompt version+ model version+ freshness requirementsWithout those boundaries, semantic caching can become a data-isolation problem.Two users can ask linguistically identical questions and still require different answers.Some Answers Should Not Be Semantically CachedBe extremely careful with:account-specific information,authorization-sensitive results,rapidly changing data,inventory,market data,legal or policy status,tool-derived live state,personalized recommendations.Semantic caching works best where equivalent questions genuinely have equivalent answers.6. Move Non-Interactive Work Off the Interactive PathAsk yourself how much of your AI workload actually requires an answer in a few seconds.Probably less than you think.Common background workloads include:document enrichment,offline extraction,nightly summaries,evaluation suites,synthetic-data generation,LLM-as-a-judge jobs,bulk classification,content tagging,migration/backfill work.Those jobs should be architected differently from chat.Several major providers currently offer substantial discounts for eligible asynchronous batch processing. OpenAI advertises 50% savings on input and output with its Batch API; Anthropic similarly documents a 50% discount for batch processing; Google's Gemini API also currently lists a 50% reduction for batch workloads. Exact availability and turnaround requirements vary by provider and model.So create two lanes. ┌── interactive lane ──> synchronous inferenceRequest/Job ─────┤ └── background lane ───> queue -> batch processingThis distinction should exist at the platform level rather than being rediscovered by every product team.Something as simple as:type LatencyClass = | "interactive" | "background" | "bulk";can become an important architecture decision.A nightly evaluation job should not accidentally pay for the same service characteristics as a user waiting on a chat response.7. Stop Paying for Output Nobody UsesContext optimization receives a lot of attention because prompts are visible.Outputs deserve the same scrutiny.Output tokens can be significantly more expensive per token than input on leading APIs. Current OpenAI GPT-5.6 Sol pricing, for example, lists $5 per million standard input tokens and $30 per million output tokens. Anthropic lists Opus 4.8 starting at $5 per million input and $25 per million output.Now inspect your UI.Does the application display the whole response?Or does it:truncate after two paragraphs?parse only a JSON field?consume only a classification label?discard reasoning-like prose?throw away five alternatives and keep one?If the consumer needs:{ "category": "billing"}don't ask the model for:Based on my careful analysis of the user's request,I believe this question is most appropriately categorizedunder billing for the following reasons...Use structured outputs where appropriate.Set output ceilings based on the task rather than one enormous global maximum.And measure tokens generated versus tokens actually consumed by the product.That ratio can be surprisingly revealing.8. Retries Need a Budget TooRetries are necessary.Unbounded retries are an architecture bug.An LLM call can fail because of:transient provider errors,timeout configuration,tool failure,schema-validation failure,content validation,application bugs.If every failure retries the entire expensive pipeline, costs multiply quietly.A sensible retry policy considers:error type+ previous attempts+ model used+ context size+ expected business valueA transient HTTP failure may justify retrying the same request.A schema-validation failure may justify changing strategy.A bad tool result may require fixing the tool input rather than asking the same model three more times.For agentic systems, use both:maximum stepsmaximum model callsmaximum tokensmaximum wall-clock timemaximum estimated spendAn agent should have a budget before it starts working.9. Make Cost a First-Class Platform ConstraintCost-aware architecture is becoming part of modern system design: model routing, caching, workload isolation, observability, rate limits, and failure budgets all influence both reliability and unit economics.The most dangerous cost optimization project is the one that succeeds once.Because six months later:a new feature bypasses routing,context grows,another tool gets added,retry limits disappear,somebody switches models,the cache hit rate falls,and the invoice quietly returns.You need mechanisms that resist that entropy.Per-Feature OwnershipEvery major LLM feature should have an owner who can see:requeststokensmodel mixcache hit rateretry ratelatencycostbusiness outcomeCost that belongs to "the AI platform" tends to become nobody's problem.Cost Regression TestingPrompt changes are code changes.Treat them that way.If an eval suite contains representative requests, a CI job can estimate:old average input tokensnew average input tokensold average output tokensnew average output tokensold success ratenew success rateold estimated costnew estimated costThen a pull request can say:Quality: +1.2%Median latency: +4%Estimated inference cost: +37%Now you can have the right engineering discussion.Maybe the quality gain is worth the cost.Maybe it isn't.The point is that the trade-off becomes explicit.Tenant and Agent Circuit BreakersFor multi-tenant platforms, budget enforcement belongs in the gateway.Think:per-user limitsper-tenant limitsper-feature limitsagent step budgetsdaily/monthly budget alertsprovider failover rulesThis is not just finance governance.It is reliability engineering.A runaway loop that happens to call an API with usage-based pricing is both an operational incident and a financial incident.A Better Order of OperationsIf I were designing an LLM cost program from scratch, I would work roughly in this order:1. AttributeKnow where the money goes.2. Remove accidental spendFix pathological retries, forgotten background jobs, duplicate calls, obsolete fallbacks, and unused generation.3. Route intelligentlyMatch model capability to task difficulty.4. Reduce unnecessary contextBudget conversation history, retrieval, memory, and tools.5. Improve cache economicsRestructure reusable context and measure hit rates.6. Separate latency classesBatch the work that doesn't need interactive latency.7. Control outputsGenerate only what downstream consumers use.8. Add regression gates and budgetsMake the savings survive the next release.This order matters.You do not want to spend two weeks shaving 100 tokens from a prompt only to discover that a retry bug was making every request three times.What I Would Not DoA few approaches sound attractive but frequently move the problem somewhere else."Just use the cheapest model."Only if your evals say it works.Cost optimization without quality constraints is product degradation."Let's summarize all conversation history."Only if you can tolerate losing information.Some history should be compressed. Some state should be structured. Some information should remain verbatim."Let's cache every answer."Caching the wrong answer faster is not optimization.Cache identity, authorization, invalidation, and freshness are part of the design."Let's reduce retrieval to three chunks."Three is not magic.Find the smallest context budget that preserves your required retrieval and answer quality."Let's self-host."Maybe.Self-hosting changes the economics from API metering to infrastructure economics: accelerators, utilization, serving software, observability, scaling, reliability, upgrades, and engineering time.At sufficiently high and predictable utilization, that can be attractive.At low or spiky utilization, the supposedly cheaper model may come with a surprisingly expensive platform.Calculate total cost of ownership rather than comparing only token prices.What Should You Optimize: Total Spend or Cost Per Outcome?This is the most important distinction in the article.Imagine usage doubles.Your AI bill grows 70%.Is that bad?Not necessarily.If successful customer resolutions grew 120%, your economics improved.Conversely, your infrastructure team could reduce the total bill by 30% while destroying response quality and user retention.That is not a successful optimization.For each meaningful feature, define a unit that connects inference to value:cost per resolved ticketcost per generated reportcost per successful workflowcost per active conversationcost per accepted code suggestioncost per qualified leadThen optimize that number.A Practical LLM Cost Review ChecklistWhen reviewing a production AI system, I would ask these questions in order:Can we attribute every LLM call to a feature and purpose?Are retries visible and bounded?Does every task need the model it currently uses?Do we have an escalation path when a cheaper model fails?How much context do we send per successful request?Which context is actually relevant?How much retrieved content reaches generation?Are irrelevant tool schemas being sent?What percentage of reusable context receives cache benefits?Are cache writes actually earning enough subsequent reads?Which jobs can tolerate asynchronous processing?How many generated tokens are actually consumed?Can an agent exceed a defined spend or step budget?Do prompt changes show cost deltas in CI?What business outcome does each dollar of inference buy?If you cannot answer the first few questions, don't start by rewriting the prompt.Start by fixing observability.Frequently Asked QuestionsDoes model routing reduce quality?It can.That is why routing should be evaluated like any other model change.Measure task success by route, define escalation conditions, and compare the resulting quality, latency, and cost against your baseline.A router without evaluation is just a cheaper way to guess.Is prompt caching always worth using?No.Its value depends on your provider's pricing model, how much context is reusable, how often it is reused, cache retention, and cache-write economics.High-reuse stable context is usually much more interesting than large one-off prompts.Should I always summarize old conversation history?No.Summarization trades token reduction for information loss.A robust pattern is often a combination of structured state, summarized older conversation, and a recent verbatim window.Is batch processing really cheaper?For eligible workloads, several major API providers currently advertise roughly 50% discounts compared with standard processing, although model support, processing windows, and feature compatibility vary.If your workload is genuinely asynchronous, it is worth evaluating.Should AI cost grow slower than usage?Ideally, unit economics improve as routing, caching, and infrastructure mature.But there is no law saying total AI spend must grow sublinearly.A company may deliberately spend more inference per request because a more capable system creates more business value.The metric to watch is not simply the invoice.It is cost relative to useful outcome.ConclusionThe expensive part of an LLM application is rarely one badly worded prompt.The cost emerges from hundreds of architecture decisions:Which model receives the request?What context gets attached?What gets retrieved?Which tools are exposed?How many times can the system retry?Can the request reuse cached computation?Does it need real-time processing?How much output can it generate?What happens when an agent refuses to stop?Those decisions eventually become the invoice.That is good news.Because it means the bill is not weather.It is not something that simply happens when an AI product becomes popular.It is a property of the system you designed.Instrument it.Route deliberately.Budget context.Cache intentionally.Batch the slow path.Bound outputs and retries.Then put enough governance around those decisions that the next feature cannot casually undo them.The goal isn't to make every token cheap.The goal is to stop paying for tokens that never needed to exist.