Most people think of an "AI bill" as one number. It's actually several stacked meters, and most teams only watch one of them.
1) Input vs. output A token is just how the model chops up text, not a whole word: roughly 4 characters, or three-quarters of a word. Every call counts two pools: tokens in (your prompt, chat history, tool instructions) and tokens out (what the model generates back), priced differently, with generation costing noticeably more than reading. On Claude models right now that gap is close to 5x across the lineup. I think of it like a taxi meter: the ride back costs more than the ride there. So the "chatty" part of an AI agent, not the context you feed it, is what drains a budget.
2) The caching meter Agent loops resend the same system prompt and tool list nearly every turn, so providers let you mark that content reusable instead of reprocessing it. On Claude, a reused cached prefix costs about a tenth of normal input price; storing it upfront costs more (roughly 1.25x for a 5-minute cache, 2x for a 1-hour one), so it pays for itself after one or two reuses. Under the hood this is what engineers call "KV caching" in inference engines: reusing already-computed attention state instead of recomputing it from scratch.
3) The reasoning meter Reasoning models add a third wrinkle: when a model "thinks" before answering (what the field now calls test-time compute), that invisible thinking text bills as output tokens too, at the expensive rate. A model that reasons a while and gives a short final answer can cost more than its visible reply would suggest.
Why this matters if you're building agents An agent isn't one call, it's a loop: read the task, call a tool, read the result, decide what's next, retry, then answer. Every turn re-sends the growing conversation as fresh input and produces fresh output, and with context windows now reaching a million tokens or more, one long-running loop is a much bigger line item than a single chat message. Without per-call, per-step accounting, you can watch the dollar figure climb with no idea whether it's one runaway retry or necessary reasoning. That visibility is what agentic AI and LLM engineering roles expect built in from day one, not bolted on after the first surprise invoice.
Quick check before you scroll: Two LLM calls use the exact same total token count (input + output combined). Why might one cost noticeably more than the other?
Full breakdown + the answer: frankduah.me/learnings/2026-08-01-token-and-cost-accounting-for-llm-apps
The answer
Because input and output tokens aren't priced the same, output tokens cost several times more per token than input tokens. A call that generates a lot of output text costs more than one that mostly just reads a big prompt, even with an identical total token count.
New here? I post a bite-size AI / ML concept like this every day, follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#AI #LLM #AIAgents #MachineLearning