When Not to Use an LLM
There's a lot of writing about what language models can do. Less about where they're the wrong tool — which is a more useful thing to know, because reaching for one in the wrong place costs you money, latency, and a class of bugs you can't reproduce.
Here are the cases where I'd push back.
When a deterministic rule already works
If the task can be expressed as a regex, a lookup table, or fifteen lines of branching logic, write the fifteen lines.
This sounds obvious and it happens constantly. Parsing a fixed date format. Validating an email. Routing a request based on a field that's already in the payload. A model will do these correctly most of the time, which is strictly worse than code that does them correctly every time and costs nothing to run.
The test: can you write down the rule? If yes, write the rule.
When you need exact computation
Models are bad at arithmetic and worse at arithmetic they're confident about. Anything involving money, inventory counts, dates that must be right, or numbers that feed downstream systems should be computed in code.
If the model is involved at all, its job is to extract the values and hand them to a function. It should not be doing the math.
When wrong answers are silent
This is the one that catches teams out.
A failing API call throws. A failing model call returns a fluent, plausible, wrong answer that passes straight through your system. If your pipeline has no human looking at the output and no way to verify it, you're accumulating errors you won't discover until a customer does.
Before adding a model call, ask what happens when it's wrong and nobody notices. If the answer is "quietly corrupted data" or "a user acts on bad information," you need verification, a human in the loop, or a different approach.
When you can't afford to evaluate it
Building the feature is the cheap part. Knowing whether it still works after a prompt change, a provider update, or a shift in your input distribution is the expensive part.
If nobody on the team is going to own an eval set, don't ship the feature. An unmonitored model call degrades silently, and the first sign of trouble is usually a support ticket.
When you have plenty of labeled data and one narrow task
If you have ten thousand labeled examples of a single classification problem, a small purpose-built classifier will likely be faster, cheaper, more accurate, and easier to reason about than a general model with a prompt.
Language models shine when you have no data and need something working today. That advantage shrinks fast once you actually have data.
When latency is the product
Autocomplete, keystroke-level feedback, anything in a tight interaction loop. If the experience only works under 100ms, most model calls are out, and designing around that late is painful.
Sometimes the fix is a smaller model, aggressive caching, or doing the work ahead of time rather than on demand. Sometimes the honest answer is that this feature shouldn't use a model.
When the cost doesn't survive contact with scale
Run the arithmetic before you build, using your realistic request volume rather than your demo volume. Per-call costs that look trivial have a way of becoming the largest line item in a service.
If the feature is only viable at low volume, that's fine — just know it now, so success doesn't become an incident.
When you need to explain the decision
Loan approvals, hiring, moderation with an appeals process, anything with a regulator or an angry customer at the end of it. "The model said so" is not an explanation, and post-hoc rationalizations generated by the same model are not evidence of anything.
Where you need auditability, use a system whose logic you can point at.
When you're using it to avoid a conversation
Some teams reach for a model because the requirements are unclear and the model seems like it can absorb the ambiguity. It can't. It just moves the ambiguity somewhere harder to see, and you find out what the requirements were when the output is wrong in a way nobody anticipated.
Vague spec plus a language model equals a vague product.
The heuristic
Reach for a model when the input is messy, the output tolerates variation, and something checks the result — a person, a validator, or a downstream step that fails loudly.
Avoid it when the rule is known, the answer must be exact, errors pass unnoticed, or the decision has to be defensible.
None of this is an argument against using these things. They're genuinely good at a set of problems that were intractable a few years ago. But the value comes from knowing which problems those are, and that's mostly a matter of knowing the other list too.
