Don't Use LLMs
Unless you really really have to!
Don’t Use LLMs
Unless you really really have to!
I’m the biggest fan of LLMs. I use them every day. But right now I spend a lot of my time trying to convince people not to use them. Why? Because you can’t guarantee availability, you can’t guarantee the outcomes, and there are much better ways of solving most problems.
Use LLMs
I spend most of my day interacting with LLMs. They can help me to research key topics, refine my positions and keep track of my progress. For one-off tasks, they are a magical tool.
Don’t Use LLMs
I was chatting with Ryan Lopopolo on X. If you haven’t read it, he wrote the excellent OpenAI piece on coding harness design. He was talking recently about the amazing improvements in long-horizon task planning in codex. I agreed, but I think that in the general case the supervisor pattern (long running agent supervising sub agents to minimize impact on context usage) is overused. You don’t need an LLM to keep track of subagents, that’s what pipeline descriptions and finite state machines using enums in a database are for.
If I don’t really know how I want a problem solved, I can spin up a SoTA model, point it at the problem and get surprisingly good results. But I find that most of my use cases are repetitive business flows, whether that is specifying, writing, testing and deploying code, or automating enrichment and outreach, or booking vendors and speakers for events.
For repetitive use cases, you are MUCH better served by having a mechanism to describe a pipeline that is run deterministically, and has model and/or human in the loop steps. It reduces token usage, improves determinism, reduces latency, heck it even saves the climate by burning less fossil fuel :) And you can always point your LLM at it so it can generate the json for you!
Two Rules
I find that if I follow two guidelines, it allows me to optimize the efficiency of my high value, repetitive business processes:
Models should be used to write code to replace themselves
Humans should be used to train models to make human review unnecessary
That’s it. If I explore a workflow in plan mode and have a supervisor run sub-agents to do something, that’s a great way to prototype a workflow and get a sense for how it should run. If I want to repeat that workflow, I need to give the agent a workflow description language so it can specify the flow and have a database keep track of the status of the runs.
If you have a step where an agent is calling an API to enrich a record and writing it to the database, is judgment actually required? If not, can the agents write a script to do it all so you’re not burning tokens for no good use?
If you have a step where an agent is classifying records based on few-shot learning, can you get it to write a simple Python rules-based script for most cases, to reduce the subset that actually requires an LLM?
And I find that as a human, if I treat all of my activity as RLHF, it doesn’t guarantee that I’ll get out of the loop, but it maximizes the chance that I’ll have to invest the minimum attention in the repetitive flows that I build, whether they are for GTM or R&D.
Drift
The temptation runs the other way. Put an LLM in the middle, let it decide the branches, summarize state, classify inputs, call tools, carry the plan. You get something running in a weekend that used to take a quarter. It’s a gift, until it isn’t.
Long-running supervisor agents drift. They stop for reasons you can’t always reproduce. They lose track of where they were. They reach for a tool they already called. They hit an edge case three hours in that they can’t recover from.
If your pipeline has to run for eight hours before it produces value, and your supervisor stops at hour six, you restart. That’s eight new hours of tokens, latency, and customer frustration. And you don’t know, on the next run, whether it will stop again.
This isn’t a problem Claude 5 fixes. The longer the horizon, the worse the math, regardless of how good the model gets.
Conditionals
Better models extend the horizons and reduce the hallucinations. But if statements never hallucinate at all.
If the only thing you need at this step is to decide whether a new ticket is a bug, a feature request, or a support question, you don’t need a frontier model. You need a classifier. If you need to decide whether an invoice is over the approval threshold, you need a comparison operator. If all you need is a conditional, why do you have an LLM?
There are dozens of these steps in any real production pipeline, and most of them are conditional logic dressed up as decision-making. The LLM can do them. It can also cost more, take longer, and occasionally invent a fourth branch nobody asked for.
Supply
Token costs could go up. They could also come down. Max plans are already going away or shrinking. What supply and demand look like next year, I don’t know, and you don’t either.
Here’s the part that matters. Price elasticity for frontier models is high. If a model gives you a 5x or 10x productivity gain on work that’s worth a lot of money, some customers will pay a lot of money to get it. That means the top of the market stays gated by willingness to pay, regardless of which direction the headline price moves.
Tom Tunguz’s recent piece on the compute crisis has the data. Nvidia Blackwell rental pricing rose forty-eight percent in two months. CoreWeave raised prices twenty percent.
OpenAI‘s CFO Sarah Friar said the quiet part out loud: “We’re making some very tough trades at the moment on things we’re not pursuing because we don’t have enough compute.”
Anthropic is in the headlines right now for reducing token availability on plans and potentially running out of capacity. OpenAI may be next. If you’re not one of the whales that the labs are wooing, good luck getting time with a sales rep (let alone customer service) for your $500k or $1m annual commit.
So here’s the question. If your foundation lab called tomorrow and told you your account was being deprioritized, what does your Monday look like?
Compile Down
That’s the whole idea. Default to deterministic code wired around the repeatable process. Keep the LLM in the loop at the specific steps where it uniquely earns its spot: classification where a rules engine is too brittle, generation of prose or code, summarization of genuinely varied input, and the occasional open-ended judgment call that nothing else can make.
Branches that can be expressed as conditionals should be conditionals.
Validation that can be expressed as schema checks should be schema checks.
Human-review stops go where the blast radius makes them worth the latency.
Let frontier models build the system. Let deterministic code run it.
Won’t Better Models Fix This?
The most common pushback goes: “If I over-engineer around today’s models, I’ll rip it all out in six months when they’re better.” Fair, for a specific kind of engineering. Scaffolding you build to patch a model that drifts, hallucinates, or forgets its instructions does become dead weight as models improve.
Deterministic business logic isn’t that scaffolding.
An if statement against a business rule doesn’t get obsoleted by Claude 5. A schema validator doesn’t become redundant. A durable pipeline step that writes the outcome to Postgres doesn’t care which model is in use this quarter. That code encodes how your business works. The model is a moving target.
The Takeaway
Use LLMs where no cheaper, more reliable, lower-latency alternative exists. Everywhere else, prototype with LLMs and then have them compile the flows down into code. While frontier access is still open, use it the way it was meant to be used. Prototype. Explore. Discover the shape of a problem you don’t yet understand. Generate the code that will run the process once it becomes repeatable.
That’s the best use of a token budget you can’t guarantee you’ll have in a year. The architecture most likely to survive the next three model releases is the one that depends on models the least.
What are you running on a model that could be running on code? I’d love to hear what you find when you go looking.