How to Hire LLM Engineers in 2026: Five Skills, Five Probes
"LLM engineer" is the fastest-growing title in software and the least standardized. The same two words cover someone who has kept a retrieval pipeline healthy under real traffic for a year — and someone who built three chatbot demos and a strong opinion about frameworks. Their CVs look identical. Both say Python, LangChain, RAG, "production experience with GPT and Claude." The difference between them is roughly the difference between your AI feature shipping and your AI feature joining the 95% of pilots that never show a return.
Interviewing for this role with a generic backend loop doesn't work, because the hard parts of LLM engineering don't show up in LeetCode or in a system-design whiteboard about load balancers. They show up in five specific competencies. Here they are, each with the probe we use to test it and the answers that separate the tiers.
The job is making a probabilistic component behave inside a deterministic system
Start with what you're actually hiring for, because the title obscures it. An LLM engineer's job is not prompting. It's owning a component that fails silently, drifts over time, bills per request and can be manipulated by its inputs — and making that component meet the reliability bar of the boring, deterministic system around it. In a normal week that looks like: reviewing eval results before a prompt change ships, chasing a retrieval regression after a corpus update, tuning a fallback path that fired too often, and explaining to finance why last month's token bill moved. Everything worth testing in an interview derives from that job description: what "AI-ready" means concretely is judgment about this component, not enthusiasm about it.
One consequence worth stating plainly: this is a senior profile. Every task above is a judgment call with production consequences, which is why a junior engineer with AI tooling doesn't substitute for an LLM engineer — the tooling generates code, not judgment. Budget accordingly, and interview for scars, not vocabulary.
Skill 1: Evals — ask how they know a change didn't break anything
The probe: "You improved the prompt and the output looks better. How do you know nothing else got worse?"
A prompt tinkerer answers with adjectives: they looked at some outputs, the responses seem better. An LLM engineer answers with a regression suite — a golden dataset of real cases, scored automatically on every prompt or model change, with pass thresholds that gate deployment. They'll distinguish between exact-match checks, LLM-as-judge scoring and human review tiers, and they'll know the failure mode of each (LLM judges drift too). Ask what's in their golden dataset: the strong candidates include the ugly cases — adversarial inputs, ambiguous requests, the ones that failed in production last quarter.
No evals story means every future change to your AI feature ships blind. This is the single highest-signal question on the list.
Skill 2: Cost — ask for a number they were responsible for
The probe: "What did a request cost in the last LLM feature you shipped, and what did you do about it?"
Inference is a per-request cost that grows with adoption, and someone on the team has to own it like COGS. Engineers who have shipped can quote the number — and, more tellingly, the levers they pulled: caching, routing easy traffic to a cheaper tier, capping output length, batching. Candidates who have never thought about cost will say the model choice was made by someone else, or that costs are an optimization for later. In production, "later" arrives as an invoice with a growth curve attached.
Skill 3: Retrieval — ask what they measured, separately from the model
The probe: "Your RAG feature gives wrong answers. Walk me through the diagnosis."
Weak candidates go straight to the prompt or blame the model. Strong candidates split the system: is retrieval surfacing the right context, measured on its own — recall against a labeled set, not vibes — or is the model mishandling good context? They'll have opinions grounded in specifics: chunking choices that destroyed table structure, hybrid search versus pure embeddings, the corpus that quietly aged out of date. The pattern to hire for is decomposition — treating the pipeline as measurable stages instead of one magic box. It's the same discipline that makes context and memory, not the model, the real bottleneck in agent systems.
Skill 4: Guardrails — ask what happens when the model fails at 3 a.m.
The probe: "The model returns malformed output, or nothing, on 2% of requests. What does the user see?"
An LLM feature without engineered failure paths is an outage generator with good demos. You're listening for the machinery: schema validation on every output, retries with budget caps, timeouts, a deterministic fallback when generation fails, and alerts keyed to quality metrics rather than just HTTP errors — because LLM failures are usually silent degradation, not exceptions. Candidates who've operated these systems talk about them the way SREs talk about runbooks; we've written about why the guardrails are 80% of the engineering in agentic systems, and the interview should weight them the same way.
Skill 5: Security — ask what an attacker's input can make the system do
The probe: "Your LLM feature can call tools. What's the worst thing a user's input could make it do?"
Prompt injection is not a theoretical concern; any system that mixes untrusted input with tool access has a new attack surface. Strong candidates reason about it structurally: least-privilege tool scopes, sandboxing, treating retrieved content as untrusted, never letting user text authorize a consequential action on its own. Weak candidates say the system prompt tells the model to be careful. If the feature you're building touches anything with money, data or hiring decisions in it, this answer matters as much as any architecture question.
The red flags that survive a good demo
Three patterns predict trouble even when the portfolio impresses:
- All demos, no operations. Every project reached "it works"; none reached month three. The gap between those states is the job.
- No numbers. Can't quote a latency, a cost per request, an eval score, an error rate from anything they built. Production leaves numbers on people.
- "The model will handle it." As an answer to failure modes, cost, or security. The model handles nothing; the harness does — and you're hiring the person who builds the harness.
- Framework vocabulary, no failure vocabulary. Fluent in the names of orchestration libraries, vague on what broke when they used them. Tools change quarterly in this field; the engineers worth hiring describe systems by how they fail, because that's the part that transfers.
What we'd put in your interview loop
- Replace one algorithm round with an evals round. Give a small golden dataset and a prompt change; ask them to tell you whether it's safe to ship.
- A cost-estimation exercise. Sketch a feature, ask for the per-request cost and the three levers they'd pull to halve it.
- A failure-path design session. The 3 a.m. question above, on a whiteboard, end to end.
- A production post-mortem conversation. Their worst LLM incident, what leaked through, what changed after. No incident stories is itself a signal.
- Reference-check the operating, not the building. Ask the reference who maintained the feature after launch — and for how long.
Titles inflate quickly in a market where AI specialists bill $200–300/hr and every generalist has an incentive to rebrand; the five probes are how you pay the premium only for the skills that earn it.
That's also, for transparency, how we vet — our CTO-designed process accepts 3% of candidates, and these five competencies are what LLM-engineer candidates get tested against, on real systems rather than take-home toys. If you'd rather start from a match than a pipeline, here's how hiring an LLM engineer through Conectia works — matched in 72 hours, vetted before you meet them.
