How to Hire an AI Engineer: A Practical Vetting Guide

Knowing how to hire an AI engineer is mostly a vetting problem, not a sourcing problem. There is no shortage of people who will tell you they build with LLMs — there is a genuine shortage of people who have run one in production, watched it fail in a way they didn't predict, and fixed it. I sit on the other side of this table regularly, so this is the process I'd run if I were hiring: how to define the role, what to probe for, a complete paid take-home with a scoring rubric, and what each engagement model actually costs.

How to Hire an AI Engineer: The Short Version

Five steps, in order:

1. Write down the specific capability you need shipped — not a generic job description with fourteen frameworks in it.
2. Screen for shipped production systems, not keyword lists.
3. Run a paid 3–4 hour take-home that mirrors your actual first deliverable.
4. Use a structured interview to probe evaluation, cost and failure handling.
5. Pick an engagement model that matches how long the work genuinely lasts.

If you only take one thing from this guide, take this filter: ask what happened to their AI feature after launch. Someone who has shipped can describe real usage patterns, a failure they found and fixed, an eval they ran, and roughly what it costs per request. Someone who hasn't will pivot to model names and framework comparisons within two sentences.

On timelines: a full-time hire typically runs four to eight weeks from job post to signed offer. An experienced contractor can start in days to a couple of weeks. On budget: expect production LLM experience to carry a clear premium over standard web development rates, because the supply of people who have actually operated these systems is thin.

One more framing, because it saves a lot of wasted interviews. Most founders I talk to don't have a hiring problem — they have a scoping problem. Once the first deliverable is written down in one sentence, the right profile is usually obvious. This guide assumes you can't code-review a candidate yourself, which is the exact reader most hiring advice quietly assumes away.

Step 1: Define the Role Before You Write the Job Post

Three genuinely different jobs all get posted under the title "AI engineer." Hiring the wrong one is the most expensive mistake in this process.

In plain terms: an AI engineer composes models; an ML engineer builds them. In my experience almost all startup AI work today is the first kind, and the sourcing pools and salary bands are different enough that confusing them wastes a hiring cycle.

The scoping exercise. Write one sentence describing the first thing this person must ship within 90 days. Something like: "A support agent that resolves tier-1 tickets end to end, with a clean handoff to a human when it can't." Then list the systems it has to touch — your ticketing tool, your docs, your auth, your database. That sentence plus that list is your job description.

Before you post it, settle whether you actually need a dedicated AI engineer or a full-stack generalist. At early stage, one person who can do both ships faster than two who each do half.

And don't omit the requirement most AI job posts forget: the feature still has to be a product. Auth, data pipelines, error states, deployment, monitoring. The same logic applies when hiring a React specialist for the surrounding app team — depth in the hard part, but not at the cost of being able to ship.

A job description skeleton worth copying:

Outcome (90 days): <the one-sentence deliverable>
Systems it touches: <auth, data sources, existing services>
Constraints: <latency target, cost per request ceiling, compliance>
You own: evaluation, cost, reliability of this feature in production
Nice to have: <two things, maximum>

Step 2: The Skills That Actually Matter

Here's the taxonomy I'd vet against, with the single question that exposes each one.

A few of these deserve more than a row.

Retrieval. Most quality problems in RAG (retrieval-augmented generation) systems are retrieval problems, not generation problems. A strong candidate reaches for better chunking, metadata filtering or reranking long before they suggest a bigger model.

Evaluation is the biggest separator. If the answer to "how do you know it got better?" is "I tried a few examples and it looked good," you have a demo builder. You want to hear about a held-out test set, graded outputs and a regression check that runs before deploys. Read a real evaluation framework first so you can calibrate what you're hearing.

Prompt design is engineering, not wordsmithing. If you're not sure what good looks like here, what prompt design work actually involves day to day is a useful reference before you run the interview.

Production hardening is where the rate is earned. Timeouts, retries, token budgets, fallbacks when the model returns garbage, cost monitoring, prompt-injection exposure. This is precisely why so many AI agent demos fail in production — the demo is the easy 20%.

What I'd explicitly deprioritize: a PhD, published papers, Kaggle rank, and the ability to name every model released this quarter.

Step 3: Where to Find Qualified AI Engineers

Every channel trades speed against signal.

The channel that works best is the least obvious one: people who have published something specific about a problem close to yours. A post or repo on retrieval evaluation is a portfolio with the reasoning attached — you get to read how they think before you spend an hour on a call.

When reading a portfolio, weight one system in production with real users above ten polished demos. Ask for a link, rough usage numbers, and what broke. If you go through an agency, confirm in writing who writes the code, whether they're dedicated or shared across clients, and where the IP and handover documentation land.

Plan for a steep funnel. I'm not aware of a reliable industry benchmark here, and I wouldn't trust one if I saw it — but in my own screening, only a small fraction of applications are worth a call, and fewer still survive a take-home. Treat that as one person's experience, not a number to budget against. Either way, inbound volume is inflated by generalists who added "LLM" to a profile recently, so the screen has to do real work.

Step 4: The Interview — Questions That Separate Shipping From Demoing

Three stages: a 30-minute screen, a paid take-home, and a 60-minute review of their own submission.

Screen questions. "Walk me through an AI feature you shipped and what it cost to run per month." "Tell me about a time the model's output was wrong in production — how did you find out?" "When would you not use an LLM for this?" That last one filters hard. Good engineers have a list.

Depth questions. How would you cut latency on a slow agent? How do you choose between a bigger model and better retrieval? What happens when a user input tries to override the system prompt? What do you log on every call?

The judgment question most interviews skip. Hand them a half-specified business problem and watch what they do. Strong candidates ask about data access, volume, acceptable error rate and who handles failures. Weak ones start drawing architecture immediately.

If you're non-technical, you can still assess depth. Insist on specificity — numbers, named failures, stated trade-offs. Ask "why not the simpler option?" twice. Check whether the explanation stays understandable as it gets deeper. People who built the thing can explain it to you; people who watched it being built retreat into jargon under follow-up.

Cheap external validation beats guessing: pay a trusted engineer for a 45-minute review of the take-home, or run a two-week paid trial. Either costs a fraction of a bad six-month hire.

The Take-Home Test I'd Use (With a Scoring Rubric)

Scope it to 3–4 hours, pay a flat fee, and make it resemble the real first deliverable rather than a puzzle.

The brief:

Build a small question-answering service over the ~50 provided documents.

Requirements

- HTTP endpoint: question in, answer + the source passages used out
- Must return "I don't know" when the corpus doesn't contain the answer
- Include an eval script with >= 10 test questions (2 unanswerable)
- Report eval results, cost per request and latency per request
- In the README: what you'd change to halve each, and what you'd do
  with another day

The shape of the deliverable, so you know what "done" looks like even if you can't read the implementation:

POST /ask  { "question": "..." }
       ->  { "answer": "...", "sources": [{ "doc": "...", "text": "..." }],
             "confident": true }

eval.py  ->  runs questions.jsonl, grades answers, prints:

  Q type          n    correct   notes
  answerable      8    7         1 missed: chunk split mid-table
  unanswerable    2    2         correctly refused
  p95 latency     1.9s
  cost / request  $0.004

The 100-point rubric:

How to read submissions. The README is worth more than the code. A 90-point submission explains why chunks are sized the way they are, includes test cases its own system fails, reports an unflattering latency number, and lists what it would fix next. A 55-point submission is a working endpoint with no eval script, no refusal behaviour, a README that says "run npm start," and no mention of cost. Both "work" in a demo. Only one tells you the truth after you hire them.

Fairness rules: cap the time, pay for it, allow AI coding assistants (then ask in the review which parts they'd have written differently and why), and never ask for free work on your real codebase.

Red Flags I See Constantly

• Keyword stuffing. "LLM, GPT, RAG, LangChain, vector databases" with no system attached to any of them, and a pivot dated to the last year with nothing shipped behind it.
• A portfolio of notebooks and demo apps. No auth, no users, no uptime, no cost story.
• Can't discuss inference cost or latency. Anyone who has run real traffic has had the cost conversation. If they've never thought in tokens per request, they haven't.
• No evaluation story. "I tested it manually and it looked good" is the most common and most disqualifying answer in this process.
• Framework-first thinking. Every problem becomes a multi-agent system, and every quality issue is answered with a bigger model or a new library.
• Vagueness under follow-up. Specifics compound in people who did the work.
• Role mismatch in either direction. Overclaiming fine-tuning and training experience for an applied role — or an applied engineer presenting as a researcher.
• Process flags on your side. An agency that won't name the engineer, a candidate who refuses any paid exercise, or someone who agrees with every architectural suggestion you make.

Cost: Full-Time vs. Contract vs. Fractional vs. Agency

Treat these as directional ranges — they vary widely by region and seniority.

The costs founders forget to budget: model API spend in both development and production, vector store and infrastructure, the time to create evaluation data, and ongoing maintenance after launch. Add those before comparing rates, or the contractor who looked expensive turns out to have been the cheap option.

A simple decision rule: if the AI work is less than a quarter of a full-time job, or the scope is one defined build, hire a contractor or a fractional lead. If it's continuous, differentiating and central to the product, hire full-time.

Where to Start This Week

Don't open a job post yet. Write the one-sentence 90-day deliverable, list the systems it touches, and decide from that sentence whether this is a 6–12 week contract build or a permanent role. Then prepare the take-home corpus — 50 of your real documents, lightly redacted — because assembling it will tell you more about your own data quality than any interview will.

If you'd rather skip the process entirely, this is the work I do. Send me the one-sentence deliverable and the list of systems, and I'll tell you honestly whether it's a contract build or a hire.

FAQ

What skills should I look for when hiring an AI engineer?

Retrieval and RAG design, agent architecture judgment, a real evaluation practice, disciplined prompt design, production hardening (cost, latency, retries, fallbacks) and solid general software engineering. Model trivia and academic credentials matter far less than a system they've run in production.

How much does it cost to hire an AI engineer?

It varies widely by region, seniority and engagement model. Independent contractors with production LLM experience command a clear premium over general web development rates, full-time hires carry salary plus equity, benefits and recruiting costs, and agencies add a markup on top. Budget model API spend and ongoing maintenance separately — they're routinely forgotten.

What's the difference between an AI engineer and a machine learning engineer?

An AI engineer composes existing models into reliable products — retrieval, agents, LLM features inside an application. A machine learning engineer builds, trains and serves models themselves. Most startup AI work today is the first kind, and hiring the wrong one is the most expensive mistake in the process.

How do I evaluate AI candidates if I'm not technical myself?

Demand specifics: real numbers on cost and latency, named failures they fixed, and trade-offs explained in language you can follow. Ask "why not the simpler option?" twice. Then pay a trusted engineer for a 45-minute review of the candidate's take-home, or run a short paid trial before committing.

What should an AI engineer take-home test cover?

A paid 3–4 hour task that mirrors your real first deliverable — typically a small retrieval-backed service that cites its sources, refuses to answer when it can't, ships with an evaluation script, and documents cost and latency per request. Score it with a published rubric so grading stays consistent.

How long does it take to hire an AI engineer?

A full-time hire usually takes four to eight weeks from job post to signed offer, assuming a screen, a take-home and a final interview. Bringing on an experienced contractor can take days to a couple of weeks, which is why many teams start with a contract build and convert later.