The Other Half of the AI Cost Equation

Walk the floor at any AI infrastructure conference this fall and you'll hear a single question asked in a dozen different locations: how do we make inference cheap enough for AI to be profitable at scale? It's the question every frontier lab and every hyperscaler is quietly obsessing over, and for good reason. The economics of running these models are brutal. Cost-per-token is the number that keeps executives awake, and an enormous amount of brilliant engineering is now pointed directly at driving it down ... better chips, tighter stacks, smarter batching, and infrastructure reconfigured from the silicon up to squeeze waste out of every request.

I have deep respect for that work. It's some of the hardest and most consequential engineering happening anywhere right now. But what that effort is forgetting is something that's largely missing from the mainstage conversation. In my opinion, I think it's the more important half of the equation ... and almost nobody is talking about it.

We are spending extraordinary energy making it cheaper to process tokens. Hardly anyone stops to ask if the data is worth tokenizing in the first place. 

The assumption hiding in plain sight

Every inference-economics discussion rests on a quiet assumption: that the data flowing into the machine is worth feeding it. Optimize the pipeline, lower the cost per unit, and profitability follows. It's a clean model. It's also wrong at the foundation, because the input is anything but clean.

In more than two decades of working with enterprise data estates, I have never once seen a "clean" one. What I see, over and over, is that somewhere between 40 and 70 percent of a typical organization's data is what we call ROT ... redundant, obsolete, or trivial. Duplicate copies of copies. Files no one has opened in nine years. Data that survived three migrations for no reason anyone can articulate. And a large share of what's left is data nobody can actually vouch for: unclassified, ungoverned, of unknown origin and unknown risk.

Now layer the inference conversation back on top. If half of what you're feeding an AI system is garbage, then a meaningful fraction of your beautifully optimized, hard-won cost-per-token is being spent processing, embedding, indexing, and retrieving material that should never have entered the system. You can make each token twenty percent cheaper and still lose, because you're paying full price on volume that has no business being there.

The cheapest token, it turns out, is the one you never process at all.

Step Zero

We call this Step Zero, and it comes before every AI initiative whether teams acknowledge it or not. Before you fine-tune, before you build a retrieval pipeline, before you turn an agent loose on your file shares ... you have to know what you actually have. What's there, where it lives, what it contains, whether it's a duplicate, whether it's a liability, whether it's worth keeping.

Most organizations skip Step Zero, not out of negligence but out of impossibility. When your data is measured in petabytes and scattered across on-prem storage, three clouds, and a dozen legacy systems, the traditional answer - copy it all somewhere central so you can inspect it - is a non-starter. Moving petabytes is slow, expensive, and risky, and by the time you've done it the picture has already changed. So, teams do the rational thing: they assume the data is fine don't bother looking at any data and move on to the exciting part.

That blind spot is why AI initiatives fail and is precisely where the money leaks out.

Understanding before optimizing

This is why I keep returning to a principle that sounds almost too simple to state: you cannot optimize what you do not understand. The entire inference-economics movement is an optimization exercise. It is, by definition, downstream of understanding your inputs. Skip the understanding and you're optimizing a system built on sand ... highly efficient at doing the wrong amount of work.

The answer is not to move the data. It's to understand it where it already lives. In-place, agentless intelligence that scans across every repository without relocating a single byte, tells you what's redundant, what's obsolete, what's sensitive, and what's genuinely valuable ... and lets you make defensible decisions before anything ever touches an inference pipeline. Eliminate the ROT. Isolate the risk. Feed your AI the fraction of your estate that actually earns its place. Do that first, and every downstream dollar the hyperscalers help you save on inference compounds, because it's being spent on data worth spending it on.

This reframes something the industry has long filed under "governance" or "compliance" ... a cost center, a box to check, work you do because a regulator makes you. I'd argue that framing is now obsolete. Understanding your data is not a compliance chore. It's an infrastructure-economics decision, and increasingly a competitive one. The organizations that get AI-ready fastest and cheapest won't be the ones with the best chips. Everyone will have access to good chips. They'll be the ones who figured out, before their competitors did, that half their data was dead weight and stopped paying to carry it.

Both halves, or neither

None of this diminishes the inference work. Both halves of the equation matter, and they multiply rather than add. Cheaper inference on a disciplined data estate is a genuine profitability engine. Cheaper inference on an undisciplined one is just a more efficient way to burn money at scale.

So as the industry gathers this fall to obsess, rightly, over token economics, my hope is that a few more people in those rooms start asking the upstream question too. Not just "how do we make each token cheaper?" but "how many of these tokens should we be generating at all?" That's the question we've been helping organizations answer, and it's the one I believe will separate the AI winners from the also-rans over the next few years.

The frontier is real, and it's worth chasing. But the fastest way forward almost always starts with going back to Step Zero ... and finally understanding what you've been carrying all along.

Eric Rodriguez is President of One Discovery, makers of Lightning IQ, the petabyte-scale in-place data intelligence platform.