Tokenomics 101

“In the beginning were the tokens, and the tokens made the world.”

When it comes to AI, it's tokens all the way down. Our appetite for them has truly exploded. Several years ago, thousands of tokens of context was the standard; now a million is typical even for open source models. And the models don’t just scale with context, they scale with time themselves. AI agents can go off for minutes, hours, even days at a time if you harness them correctly. It gets crazier, as AI agents can spawn up their own subagents too. A particularly famous / infamous recent example is Jarred Sumner’s rewrite of Bun in Rust, which at peak ran four Claude Code workflows at once, each with sixteen Claudes inside it — about 64 Claudes at a time. Absolute bleeding edge, unthinkable a few years ago, but it is the reality we live in. The takeaway: the world has an insatiable thirst for tokens, and it’s not slowing down.

So now, we’re left cleaning the bill, figuring out how to deal with this, knowing that this is only going to grow.

The fundamentals

Before we can trim token bills, let’s take a step back and define exactly what we’re paying for. Virtually all major providers price tokens in three ways: input, output, and cached tokens. Input and output are self-explanatory, while cached tokens represent sending the call to a server that recently processed the same prompt, which can save enormous compute and time. That last one is the interesting one. Caching is a core, fundamental computing principle, and the discrete turn-by-turn nature of LLM conversations makes them naturally well suited to it. That said, it's mostly taken care of automatically these days behind the scenes, so we won't focus on it too heavily.

Given the exploding costs, how can we measure and size everything up appropriately?

As LLMs went from chatbots to agents, how we measure their progress and the work they do has also changed. These days, the most effective way to contextualize token pricing is cost per successful task. Instead of thinking per conversation, per tool call, think bigger, and realize that agents that can go out and solve tasks over a long period of time should be measured accordingly.

Optimizations

Given what we talked about, how can we actually manage the exploding costs? Beyond the obvious, such as caching and compaction, which are mostly automatic these days, the levers fall into three buckets: model routing, tooling, and capacity planning.

Model routing

The first is understanding exactly what the model you’re dealing with is designed for, and what it can do. Every single AI model behaves differently, and was trained differently for different tasks. Don’t let a trillion-parameter model center a div for you – it can sit and think for minutes wasting time. Right-size the model, then right-size the actual thinking it does. For a particular example, check out our hosting of OTel 2.0, a specialized lightweight model designed for the telecom industry that punches far above its parameter class, saving time and money!

What’s really going on is a routing problem: a very hot topic, and something we’re deeply familiar with here at Featherless, as we host over 40,000 open source models, with the mission to make them as accessible as possible. One thing you may want to consider is ways to route simple tasks to easier models, which is a pretty common pattern, but take note that some things don’t have to be in a literal “switch to a different model” way. For example, the concept of “Mixture of Experts”, which is really routing under the hood: a trillion-parameter model might only activate 50 billion of them per token. Routing is everywhere even when you don’t see it.

Tooling

But how do we provide the models themselves, that are smart enough to make decisions autonomously, with the tools they need to efficiently do so? For this, we turn to tooling, in particular, harnesses. Virtually all major models released today can solve some of the hardest math and computer science puzzles right out of the box, but harnessing them for long-context problems, and doing it efficiently, is a completely different story. To manage your agentic costs, you’ll want a lightweight harness. A very popular example these days is Pi, which is designed to be minimal from the start (Pi’s system prompt is only 200-1000 tokens), giving the model a small set of basic tools such as read files, edit files, run commands, and then mostly getting out of the way. The point isn’t to build some crazy agentic framework around the model, rather you want to give enough tooling to actually go out and do useful work on its own. Ultimately, simple scaffolding means fewer tokens per task, and fewer tokens per task means a smaller bill.

Another area worth looking at is memory – particularly in the context of organizations, of which there’s no shortage of startups working to tackle this problem. Memory can be anything from shared SKILLS.md files to a global company-wide library. Ultimately, the core point of memory is to save time looking up information, filling up the context, thus saving time and tokens. Solutions vary wildly depending on the complexity and scope of the memory. At a fundamental level, memory short-circuits the "discovery phase" when an AI interacts with a codebase, which is exploring and re-learning an agent would otherwise redo on every task. It is critical for saving tokens.

Capacity planning

The final subject we’ll discuss is tradeoffs between speed and cost. We talked before about how agents can do tasks asynchronously. But what if the task isn’t urgent, or maybe you’ve configured it to happen while you sleep? In the same way some energy grids charge less during off-peak hours, we foresee a rise in slower tokens per second usage in exchange for cheaper pricing. This is a great way to save costs on repetitive, predictable tasks that you’ve scoped appropriately, and it will become more attractive as agents get even more powerful and are able to handle more difficult routine tasks on their own.

Dedicated instances

Sometimes the above optimizations just aren’t enough. At Featherless, we can help you deal with these exploding costs. Get set up today with a Featherless dedicated GPU instance, your own personal reserved GPU infrastructure for serious production AI workloads, designed from the ground up to handle whatever you need.

We give you full GPU access at a flat monthly rate so you can let your AIs run at full throttle, all day and night. For extreme traffic spikes, you’d fall back to pay-per-token capacity. Because your own GPU stays busy, it works out far cheaper per token than paying per-token for everything.

We have GPUs, and you have problems that need to be solved. Find out more here, give us a call and we’ll help tackle your biggest AI problems.

Reserve dedicated GPU capacity