When to Use an LLM: How a Forward Deployed Engineer Decides

When should you use an LLM? The short answer most people don't want to hear: not yet. Reach for a large language model only after the cheaper layers — hardcoded rules, structured data, plain search — provably can't represent what your user is asking, and only once you've built a data foundation solid enough for the model to stand on. An LLM is the most expensive, least predictable tool in the box. As a forward deployed engineer, my job isn't to ship AI. It's to ship the cheapest thing that solves the customer's problem — and to know exactly when that thing has to become an LLM.

This post is the decision rule I actually use, built up from a small worked example and then stress-tested against a real project where I got the order wrong.

#The default answer is "not yet"

The instinct in 2026 is to start with the model. There's a prompt for everything, so why not pipe the user's request straight into one and ship?

Because every layer of intelligence you add has a price. A hardcoded lookup is instant, free, and deterministic — you can read it and know exactly what it will do. A database query is nearly as cheap and just as predictable. An LLM call is none of those things: it costs money per request, adds seconds of latency, and will occasionally return something confidently wrong. Those tradeoffs are sometimes worth it. They are not worth it for a problem a dictionary could solve.

This isn't an anti-AI position — it's an engineering one. The teams who get the most out of language models are usually the ones who reserve them for the work nothing cheaper can do. (Explosion's "Against LLM maximalism" makes this case well: wrap the model in conventional structure rather than handing it the whole problem.) The skill is knowing where that line is. The easiest way to find it is to build up to it.

#A question generator, built one layer at a time

Take a small product: a tool that reads a "theory bank" — notes on a topic — and turns it into practice questions. Here's how I'd build it, and crucially, why I'd move from each version to the next. The trigger for moving up a layer is the whole point.

Layer 1 — hardcoded data. Start with the dumbest possible version: a dictionary mapping topics to questions.

QUESTIONS = {
    "python": ["What is Python?", "What is a generator?"],
    "javascript": ["What is let?", "What are closures?"],
}

The user picks a topic, you return the list. No AI, no database — and for a fixed set of topics, this is genuinely enough. Ship it. Trigger to move on: users want "the easy ones first."

Layer 2 — add structure. A flat list can't express difficulty, so give each question a field:

QUESTIONS = {
    "python": [
        {"question": "What is Python?", "difficulty": "easy"},
        {"question": "Explain the GIL.", "difficulty": "hard"},
    ],
}

Now you can filter by level. Still no AI — just better-shaped data. Trigger to move on: users stop picking from menus and start typing.

Layer 3 — add search. Someone types "difficult questions on Python." You don't need a model to handle that; you need to map words to your existing structure with a few keyword lists:

DIFFICULTY_HINTS = {
    "hard": ["difficult", "hard", "complex", "advanced"],
    "easy": ["easy", "simple", "basic", "beginner"],
}

Match the topic and the difficulty hint out of the input, then run the same filter as before. This handles a surprising amount of real-world phrasing — and it's still deterministic and free. Trigger to move on: the input becomes genuinely open-ended, and the "bank" gets too large and unstructured for keywords to interpret.

Layer 4 — now the LLM earns its place. Here is the threshold. When the user's request can't be enumerated in rules — "give me three questions that connect generators to memory usage, harder than what I've already seen" — and the source material has to be interpreted rather than looked up, keyword matching falls apart. This is the job an LLM is actually good at: reading unstructured context and generating something new from it.

Notice what the model replaces. In the earlier layers there was an implicit human in the loop — the person who read the theory and wrote good questions. The LLM is finally justified because it's doing that job, not a job a dictionary could do.

The implementation is a prompt, not magic:

You are a teacher. Your job is to generate practice questions.

Student request: {user_input}
Generate questions from this material: {question_bank}

RESPONSE FORMAT: ...
RULES: ...

Layer 5 — give it memory. Once it's generating, follow-ups need context: "harder than the last set" only means something if the model can see the last set. Add prior questions and answers to the prompt. Same model, more context.

#The rule this reveals

Read those five layers back and the decision rule falls out on its own:

Escalate to the next layer — and to the LLM last — only when the cheaper layer provably can't represent the input.

Not when the LLM could do the job. It almost always could. You escalate when the cheaper layer demonstrably can't: when the input stops being something you can enumerate, and the source stops being something you can index. Everything below that line stays rules, structure, and search — because those are cheaper, faster, and you can actually debug them.

If the worked example were the whole story, this is where I'd stop. But the clean ladder is the ideal. Real projects are messier — and the mess is where the real lesson lives.

#Real life isn't a clean ladder: a story from the field

On a project at Numerize, we were building something to extract and surface financial data from messy sources. The first version was the cheap one: a script pulled the data from various services, and we rendered it as a dashboard.

The clients weren't interested. The dashboard showed them static numbers, but what they actually had was questions — they wanted custom cuts, comparisons, and answers the fixed views didn't offer, and they couldn't ask any of them. The cheap layer had hit its real limit, not an imagined one. So far, so much like the ladder.

Then I climbed too fast. We stuffed the full context into a prompt and bolted on an LLM. When that strained, we reached for the textbook answer — RAG — to feed the model the right slices of data. It didn't work. Financial data resists naive retrieval: we never worked out a chunking strategy that held up, because the meaning lives in the relationship between a number, its label, its row, and a rule in a footnote. Chunk it and you shred exactly the context that made it answerable, so retrieval couldn't reliably put the right numbers in front of the model.

What actually worked was going back down the ladder. We scraped the data properly, processed it, and inserted it into a structured database — the foundation we'd skipped. Then we put the Claude SDK on top and built a full agent over that clean, queryable layer. The same model that failed on raw retrieval succeeded once it was standing on structured data.

That's the lesson the tidy example hides: an LLM amplifies a foundation you've earned; it doesn't replace one. RAG didn't fail because retrieval is bad. It failed because we asked the model to do the data engineering we hadn't done yet. The order isn't always a clean climb — sometimes you overshoot, and deciding well means being willing to walk back down.

#So when should you use an LLM?

Putting the example and the scar tissue together, here's the actual checklist.

Use an LLM when:

• The input is open-ended natural language you can't enumerate with rules.
• The task is interpretation or generation — reading messy context and producing something new — not exact lookup or arithmetic.
• The mapping between input and output is too large or too fuzzy to maintain by hand.
• You already have clean, structured data underneath it.

Don't reach for an LLM when:

• A lookup, a rule, or a keyword search already solves it.
• You need determinism, auditability, or guaranteed-correct math.
• You haven't built the data foundation yet. Fix that first — the model can't.

#Why this judgment is the forward deployed engineer's job

This is the part that doesn't show up in a tutorial. A forward deployed engineer is measured by the outcome the customer actually got, not by the sophistication of what got shipped. That makes "when to use an LLM" a daily judgment call, not a default. You sit close enough to the problem to know that the clients didn't want a dashboard — they wanted answers. And you stay close enough through the failures to notice that the model isn't the hard part; the foundation under it is.

The cheapest tool that solves the problem wins. Often that's a dictionary. Sometimes it's an agent. Knowing the difference — and being honest when you've guessed wrong — is the job.