Hiring an LLM engineer: how to tell production experience from a weekend demo

An LLM demo takes a weekend. Paste an API key, send the user’s question to a model, stream the answer into a chat bubble. It looks like a product.

The real work starts the week after launch. Answers are wrong in ways nobody can reproduce. The monthly API bill arrives. A provider deprecates the model you built around. A paying customer’s request times out halfway through a response, and nobody wrote code for that.

When you hire an LLM engineer, you are paying for someone who has already lived through that week. The trouble is that résumés for both kinds of people now look identical. Here is how to tell them apart.

What production LLM work actually involves

Five areas show up in almost every real AI feature. A candidate with production experience will have opinions about all of them.

Retrieval quality. If your product answers questions from your own documents, most wrong answers are retrieval problems, not model problems. Chunking strategy, metadata filters, hybrid search, and re-ranking matter more than which model you call.

Evaluation. How do you know a change made things better? Production engineers keep a fixed set of real questions with known good answers and score every prompt or model change against it before it ships. Without that, every “improvement” is a guess.

Cost and latency. Caching repeated calls, routing easy requests to smaller, cheaper models, trimming context, and setting token budgets. These decisions can change your unit economics dramatically.

Failure handling. Timeouts, rate limits, malformed output, provider outages. What does the user see? Is there a fallback model? Are partial responses handled?

Model churn. Providers ship new models and retire old ones constantly. Good engineers pin versions, keep prompts and model choice behind a thin abstraction, and re-run evals before switching.

Questions that expose the difference

You do not need to be an AI expert to ask these. Listen for specifics.

“How did you know your answers were getting better or worse?” Strong: “We had around a hundred real user questions with expected sources. Every change ran against them and we tracked the score.” Weak: “We tested it manually and it seemed better.”

“What happened to your costs after launch, and what did you change?” Strong answers mention a specific fix: caching, a smaller model for classification, shorter context. Weak answers have never looked at the bill.

“Tell me about the last time a provider changed or retired a model you depended on.” Strong: they describe re-running evals, a prompt that broke, and how they rolled out the switch. Weak: it has not happened to them, which usually means nothing they built stayed live long enough.

“A request times out halfway through a response for a paying user. What happens?” Strong: retries with backoff, a fallback model, a clear message to the user, logging for follow-up. Weak: a blank stare, or “that doesn’t really happen.”

“When would you not use an LLM for this?” The best engineers push back. Some problems are better solved with a regular expression, a database query, or a rules engine, and they will say so.

A trial task worth doing

If you want to see the work rather than hear about it, a small paid task tells you more than any interview:

1. Give them 20–30 of your real documents (after an NDA) and 15 questions users actually ask.
2. Ask for a basic retrieval pipeline and a scored evaluation of those 15 questions.
3. Ask for a short written note: which questions failed, why, and what they would try next.

The note is the most valuable part. It shows whether they debug systematically or just swap prompts until something works.

A real example: models changing weekly

Magicshot.ai runs an AI creative studio with 85+ tools for image, video, and audio. Model APIs change constantly, and every change risks breaking credit billing or the community gallery.

We placed full-stack and LLM-focused engineers who had already shipped inference in production. Their trial week was not a sandbox exercise. It was a live integration: new model in, billing intact, gallery still serving. Four placed engineers now own the generation pipeline, credits, and dashboard month-to-month, while the founders focus on partnerships and growth. The platform serves 500K+ creators and has generated more than 50 million images.

Job titles are a mess, so hire for the work

“AI engineer,” “LLM engineer,” “ML engineer,” and “AI developer” get used interchangeably. A rough guide:

• LLM engineer: builds on existing models. Retrieval, prompts, evals, cost, reliability. See hiring an LLM engineer.
• AI agent developer: LLM engineer who specializes in tool-using, multi-step workflows. See AI agent developers.
• Machine learning engineer: trains and serves models, owns data pipelines and training infrastructure. See ML engineers.

If your product calls models from Anthropic, OpenAI, or Google rather than training its own, you almost certainly want the first.

Cost

LLM engineers on Hiring.co run $500 to $2,400 per week depending on seniority and hours. Billing is month-to-month, with a 30-day no-commitment trial and a replacement guarantee if a hire does not perform.

If your AI feature works in the demo but not in front of customers, book a call. Bring the failure cases. They are the most useful thing you can show us.