Every AI team hits the same wall around month four. The demo was magic, the pilot went live, and then the invoice showed up. The first instinct is always the same: downgrade the model. Move to something smaller, cheaper, faster.
Here is the part most teams miss: you are almost never paying for intelligence. You are paying for waste.
A language model charges you by the token - roughly, by the word - for everything you send in and everything it sends back. Nobody bills you for "one question." They bill you for the size of the envelope. And most production systems are shipping a novel when a postcard would do.
We rebuilt that envelope. Same model. Same quality bar. Roughly 80% off.
Where the money actually goes
Before the fixes, one honest look at a real request.
A typical agent call in production carries: a bloated system prompt that three engineers have appended to and nobody has pruned, 12 retrieved documents when 2 were relevant, the entire conversation history replayed on every single turn, and an answer that rambles for three paragraphs when the downstream system only reads one field of it.
That last one stings the most, because output tokens typically cost four to five times what input tokens cost. Verbosity is the most expensive habit in AI engineering.
The five moves -
1. Put the prompt on a diet. We audited what was actually being sent. Half the system prompt was instructions for edge cases that occur in under 1% of calls now loaded conditionally, not universally. Retrieval was fetching twelve chunks; re-ranking and sending the top three improved accuracy and cut input by 60%. Fewer, better tokens beat more tokens. Consistently.
2. Make the model shut up on purpose. We capped response length, forced structured JSON output instead of prose, and wrote prompts that ask for a decision, not an essay. Average output dropped from ~400 tokens to ~180. Because output is the expensive side of the ledger, this single change is usually the highest-return hour of work available to any team.
3. Stop paying for the same sentence twice. Most providers now offer prompt caching: if the opening block of your request is identical to last time, you pay roughly a tenth for it. The catch is that the stable content must sit at the front of the prompt, byte-identical. Most teams accidentally break this by injecting a timestamp or a user ID at the top. Reorder: static instructions first, variable context last. At an 80% cache-hit rate, that stable block becomes nearly free.
4. Don't call the model at all. This is the uncomfortable one. A meaningful share of production traffic does not need a language model. "What are your OPD timings?" has one answer, forever. A cached response, a lookup, or a small intent classifier handles it in milliseconds for zero cost. We deflected around 15% of calls before they ever reached the model and users noticed only that things got faster.
5. Batch what nobody is waiting for. Nightly summaries, quality audits, transcript scoring, evaluation runs none of these have a human staring at a loading spinner. Most providers discount asynchronous batch jobs by about 50%. We moved every non-realtime workload over and pocketed the difference.
Why this matters beyond the invoice -
For clinicians: cost discipline is what makes it viable to keep the strong model on the clinical path. Every rupee saved on FAQ traffic is a rupee that funds the model actually reading a patient query. Cheap-everywhere is not safe. Cheap-where-it-doesn't-matter is.
For investors: gross margin in AI products is not determined at the model-selection meeting. It is determined by the token architecture underneath. Two companies on the identical model can run 4x apart on unit economics. When you evaluate an AI business, ask what a single transaction costs — and whether that number has moved in the last two quarters. If the team can't answer, they aren't measuring it.
For builders: instrument cost per request before you optimize anything. You cannot fix a bill you cannot see. Every team we know that tracked cost-per-conversation on a dashboard found something embarrassing within a week — a retry loop, a duplicated tool call, a prompt that quietly tripled in size after a hotfix.
The fastest way to make AI affordable isn't a cheaper model.
It's a shorter conversation
