Keeping an Eye on Token Costs
Language models behind a cloud API charge per token. That's convenient, because you don't have to buy a GPU or run a server – but it's also the most opaque cost center I've ever had. A bill at the end of the month gives you one number. It doesn't tell you which job is eating the money, which model is expensive, or whether usage is quietly creeping upward.
So I started measuring. Not with an off-the-shelf tool, but with a small Python script that pulls the API's usage data and writes it to a file. A second script turns that into per-job token counts. A third looks at the history. Together they're maybe 200 lines – and they've told me more about my costs than any bill ever did.
What I measure
Three numbers per model and per job: input tokens, output tokens, and the price derived from them. Prices differ a lot between models, and a big model with a long context isn't simply "better" – it's mostly more expensive. If you don't break usage down, you only see the total and can't tell whether it's justified.
The per-job breakdown was the real eye-opener. A single scheduled job – a heartbeat that regularly checks the state of my services – accounted for almost half of my total usage. Not because it's complicated, but because it ran often and dragged a long context along with it every time.
What I changed
Three levers, in this order:
- Lower the frequency. The heartbeat ran every 15 minutes. It now runs hourly. The information it provides is just as valuable after an hour as after a quarter of an hour – the usage is a quarter.
- A smaller model for routine work. Not every task needs the biggest model. For simple, recurring jobs a smaller, faster model with a large context window is enough. The switch cut those jobs' usage by more than half without making the results worse.
- Alert instead of report. A daily check only pings me when usage crosses a threshold. Silence means everything is within bounds. A weekly report sums up the numbers so I see the trend without having to look at it every day.
What I learned
The most important sentence is the same one as with monitoring: measure first, then optimize. Before measuring, I couldn't have said which job was expensive. I would probably have saved in the wrong place – on the model I rarely use, instead of the job that runs constantly.
The second point: token usage is not a static quantity. A job that's frugal today can become expensive tomorrow because its context grows or because someone attaches a bigger model to it. That's why tracking is a scheduled job, not a one-off project. The scripts run on their own, and I only look at the outliers.
The third point is the most uncomfortable one: the biggest cost source wasn't a bug in the code, but a decision I made myself – a job that ran too often, with a model that was too big. Measuring also means watching yourself save money. That's the part no tool does for you.