Keeping an AI feature affordable at scale
Most AI bills are not caused by traffic. They are caused by sending the same context over and over.
Find where the money actually goes
Before optimising anything, log tokens per request broken down by feature. Teams are routinely surprised: the expensive path is rarely the one they worried about.
The four levers, in order of payoff
Stop resending static context. A long system prompt, a product catalogue or a policy document sent on every single call is usually the largest line item. Cache what does not change between calls.
Shorten the retrieved context. Sending twenty documents when three would do is a habit, not a requirement. Measure answer quality as you cut. Most features hold up on far less context than they were shipped with.
Route by difficulty. Not every request needs your best model. A cheap classifier deciding "easy or hard" in front of two models often cuts cost substantially with no visible quality change.
Cache whole answers. In commerce, a small number of questions cover a large share of traffic. Shipping times, return policy, size guidance. Those do not need a model call at all after the first one.
The trap to avoid
Do not optimise cost before the feature is correct. Every change above trades some quality for money, and you cannot see what you traded away if you never had a working baseline. Get it right first, measure, then make it cheap.
ToolGo Team
Product & Engineering
We build practical SaaS tools and write about what we learn — from product decisions to technical deep-dives.
