AI Engineer Interview Prep: Questions, the Loop, and a Mock Test
AI engineering is a young enough job that the interview for it is still being invented, and the firms doing the inventing have started publishing their reasoning. This page uses that public material rather than folklore: how the stages are described by companies that document them, the explicit rules that now exist about using AI during your own interview, and why technical screens are being rebuilt from scratch. Then it checks your knowledge across the five areas the questions cluster around. Sources are listed at the end with the date each was read.
AnyLearn is independent and is not affiliated with, endorsed by, or sponsored by any company named on this page. Companies are named only as the source of publicly published statements about their own hiring.
How the process is structured
| Round | Length | What it tests |
|---|---|---|
| 1.Experience and motivation screen[3] | Not published | Background, motivation, and your own questions. Anthropic describes it as: "We'll ask about your experience and what motivates you, and you'll have time to ask us about Anthropic." It also states all its interviews are conducted remotely over Google Meet. |
| 2.Live coding, with lookups allowed[3] | Not published | Writing, running and debugging code in a shared environment. Anthropic names "live coding tools like Colab and CodeSignal" and says you "can look things up," while expecting enough fluency in basic syntax and standard libraries that looking things up does not consume the session. |
| 3.Take-home or applied problem[2] | Not published | An applied problem deliberately sized so that strong candidates do not finish. Anthropic's published design principles are that it should be representative of the job, produce a wide scoring distribution, avoid gatekeeping on specifics, and carry "enough depth that even strong candidates don't finish everything." |
Bracketed markers point to the dated sources at the end of this article. Loops change; check the retrieval dates before relying on a round count.
The job the interview is testing for
An AI engineer builds systems around models rather than building the models. The work is retrieval, evaluation, tool use, latency and cost, failure handling, and knowing which of those is the actual bottleneck. That is closer to backend engineering with an unusual dependency than it is to machine learning research.
This matters for preparation because the two loops look different. A research interview probes optimisation, architectures and statistics. An engineering loop probes whether you can make an unreliable component behave acceptably inside a product.
Anthropic, which publishes more about its process than most, is explicit that the boundary is porous and that you should pick a side anyway: "engineers here do lots of research, and researchers do lots of engineering. If you have an engineering background, apply as an engineer, you'll perform better in the interviews." It also states that "About half our technical staff had no prior ML experience," which is a direct signal about what the loop is filtering for.
Key idea: The interview is not checking whether you can derive backpropagation. It is checking whether you can make something unreliable useful.
Using AI during your own interview, which is now an explicit rule
This is new, it is now written down, and candidates get it wrong in both directions. Anthropic publishes a stage-by-stage rule:
| Stage | The rule |
|---|---|
| Application | Allowed, with a constraint: "Please create your first draft yourself, then use Claude to refine it." |
| Interview preparation | Allowed: "Use Claude to research Anthropic, practice your answers, and prepare questions for us." |
| Take-home assessment | Not allowed by default: "Complete these without Claude unless we indicate otherwise," because "We'd like to assess your unique skills and strengths." |
| Live interview | Not allowed by default: "This is all you, no AI assistance unless we indicate otherwise," because "We're curious to see how you think through problems in real time." |
The principle they state is "Use AI to refine your ideas, not replace them."
Two practical consequences. First, the default is not uniform across the loop, so read the instructions for each stage rather than assuming one policy. Second, where lookups are permitted they are genuinely permitted: Anthropic says candidates "can look things up, just be comfortable with basic syntax and standard libraries so it doesn't eat up your time." Fluency still gets measured, just as speed rather than as recall.
Why the technical screen keeps being rebuilt
The most useful public account of how AI has changed technical interviewing comes from Anthropic's engineering blog, describing three versions of the same performance engineering take-home. The original asked candidates to optimise code for a simulated accelerator. It was defeated by Claude Opus 4, redesigned, then defeated again by Claude Opus 4.5. Each time the fix was a redesign rather than a difficulty increase.
The current version uses "a tiny, heavily constrained instruction set, optimizing solutions for minimal instruction count," modelled on Zachtronics puzzle games, and candidates are explicitly allowed to use AI assistance. The design shift is stated in one line: "The original worked because it resembled real work. The replacement works because it simulates novel work."
The stated principles for a good problem are worth reading as a description of what you will be scored on:
- Representative. "The problem should give candidates a taste of what the job actually involves."
- High signal. Avoid "problems that hinge on a single insight," and aim for a "wide scoring distribution."
- No gatekeeping. "People with good fundamentals can learn specifics on the job."
- Deep. "Enough depth that even strong candidates don't finish everything."
That last one is the single most useful thing to internalise. If the problem is designed so nobody finishes, then not finishing is not the failure mode. Stopping to explain a trade-off is not lost time, it is the measurement.
The five areas the questions cluster around
Across published descriptions and the shape of the work itself, AI engineering questions concentrate in five places. The mock below is built on them.
Retrieval. Chunking, embedding choice, hybrid lexical and dense search, reranking, and diagnosing whether a bad answer came from retrieval or from generation.
Evaluation. Golden sets, LLM-as-judge and its failure modes, regression detection, and why an offline score that does not move can still be hiding two effects cancelling out.
Agents and tool use. Compounding per-step error, tool selection accuracy, structured calls, and where to put a constraint so it actually binds.
Guardrails. Prompt injection as a data-path problem rather than an input-validation one, and the difference between asking a model not to do something and not granting it the capability.
Serving, latency and cost. Prompt caching, quantisation, streaming, and which metric a user actually perceives.
Notice what is not on the list: training loops, loss functions, and architecture design. Those belong to the research interview.
What they actually ask
1.How would you decide between retrieval and fine-tuning for a customer support assistant?
What a strong answer coversFrame it as what each one actually changes. Retrieval changes what the model knows at request time and is the right tool when the knowledge is large, changes often, or needs to be cited. Fine-tuning changes behaviour, format and tone, and is the right tool when you need consistent structure or a house style that prompting keeps drifting away from. Strong answers point out that the two are not alternatives for the same problem, note that support content changes weekly so a fine-tune would be stale immediately, and mention the operational cost difference: a new document is an ingest job, a new behaviour is a training run and an eval cycle.
2.Your retrieval system returns confident answers that are not supported by the retrieved documents. How do you debug it?
What a strong answer coversSplit the pipeline before theorising. Retrieval and generation fail differently, and the first measurement is whether the right passage was in the context at all. If it was not, the problem is chunking, embedding choice, or the absence of lexical matching for rare tokens. If it was, the problem is generation: the prompt does not constrain the model to the context, or there is no instruction to abstain. Strong answers mention measuring both separately, adding a groundedness check that compares claims to retrieved spans, and giving the model an explicit way to say it does not know, since a model with no abstain path will always produce something.
3.How would you evaluate an agent that takes multi-step actions, where there is no single correct output?
What a strong answer coversSeparate outcome from trajectory. Outcome evaluation asks whether the end state is correct, which is checkable when the task has a verifiable result such as a booking made or a file written. Trajectory evaluation asks whether the steps were sensible, which usually needs either a rubric applied by a judge model or human review of a sample. Strong answers note that per-step success compounds, so a step-level metric is worth tracking separately from end-to-end success, and that you want a small set of deterministic tasks with checkable outcomes as a regression suite even if most real usage is open-ended.
4.You need to halve p95 latency on an LLM feature without losing noticeable quality. What do you try?
What a strong answer coversMeasure where the time goes before changing anything: retrieval, queueing, prefill on a long prompt, or decode. The fixes differ. A long unchanging system prompt is a prompt caching problem. A long output is a decode problem, addressed by streaming, shorter formats, or a smaller model. Many sequential model calls are an architecture problem, addressed by parallelising independent calls or removing a step. Strong answers distinguish perceived latency from total latency, note that streaming changes time to first token rather than completion time, and say plainly that routing easy cases to a smaller model is usually the largest single win.
5.How do you stop a tool-using agent from doing something destructive?
What a strong answer coversThe key move is recognising that a system prompt is a request, not a control. Capability should be constrained where it is granted: do not give the agent a delete tool, or scope its credentials to a namespace, or require an explicit human confirmation for irreversible actions. Strong answers separate reversible from irreversible operations and treat only the second class as needing a gate, mention that retrieved content can carry instructions so anything the agent reads is untrusted input, and note that logging every tool call matters because the failure you have not imagined is the one you will need to reconstruct.
6.A feature has just launched and there is no labelled data. How do you build an evaluation set?
What a strong answer coversStart with real traffic rather than synthetic data: sample actual queries, stratify by type so rare-but-important cases are represented, and hand-label a few dozen. Fifty carefully chosen examples that cover distinct failure modes beat a thousand near-duplicates. Strong answers describe seeding the set from the cases that already went wrong, adding each new production failure as a permanent case, and using a judge model only after checking its agreement with human labels on a subset. They also note the point of the set is regression detection when the prompt, model or retriever changes, not proving the system is good.
Three sample questions, answered
These three show the level the mock is pitched at, with the answer and the reasoning in the open. The graded paper keeps its answer key server-side.
- The model's temperature setting
- Whether retrieval returned the right passages at all, before blaming generation
- The vector database's replication configuration
- The maximum output token limit
Why: Retrieval and generation fail differently and need separate measurement. If the correct passage never reached the context, no amount of prompt work will fix it. Checking retrieval first tells you which half of the pipeline you are actually debugging.
- Ship it, since nothing got worse
- Increase the size of the evaluation set tenfold
- Break the score down by query type, since a flat average can hide one segment improving while another degrades
- Switch to a larger generation model
Why: An unchanged aggregate is not evidence of an unchanged system. Two opposing effects of similar size cancel in the mean, which is exactly the case where a segment-level breakdown reveals a real regression you were about to ship.
- 90 percent
- 48 percent
- 63 percent
- 10 percent
Why: Independent per-step success rates multiply, so 0.9 to the power of 7 is about 0.48. This compounding is the single most important arithmetic fact about agent design: it is why shorter chains and verification steps matter more than a better prompt.
An 18-question knowledge check
This is a knowledge check, not a simulation. The real loop happens on a whiteboard, in an editor, and in conversation. What this paper does measure is the underlying knowledge those rounds draw on: each question is tagged with a topic, grading happens per topic, and a weak topic points you at the course that fixes it.
- 1.Why can chunking documents too coarsely hurt retrieval quality?Retrieval and RAG
- 2.What does a cross-encoder reranker add on top of vector retrieval?Retrieval and RAG
- 3.Why do production systems often combine dense embedding search with BM25 keyword search?Retrieval and RAG
- 4.HNSW is an approximate nearest neighbour index. What is the practical trade-off it makes?Vector search and indexing
- 5.When do cosine similarity and dot product produce the same ranking of results?Vector search and indexing
- 6.Why does switching to a different embedding model force you to reindex your whole corpus?Vector search and indexing
- 7.What is the main risk of using the same model as both generator and evaluator?Evaluation and monitoring
- 8.What is a small hand-labelled golden set most useful for?Evaluation and monitoring
- 9.Why is offline evaluation insufficient on its own once a feature is live?Evaluation and monitoring
- 10.Why does a multi-step agent become less reliable as the number of steps grows?Agents and tool use
- 11.Why is a small set of well-described tools usually better than a large one?Agents and tool use
- 12.What does requiring an agent to emit a structured tool call, rather than free text, primarily buy you?Agents and tool use
- 13.Why is prompt injection different in kind from an ordinary input-validation bug?Guardrails and red-teaming
- 14.Where is the most reliable place to enforce that an agent cannot delete production data?Guardrails and red-teaming
- 15.What is red-teaming an LLM feature primarily for?Guardrails and red-teaming
- 16.How does prompt caching reduce cost?Serving, latency and cost
- 17.What does quantising a model from 16-bit to 4-bit weights primarily buy you?Serving, latency and cost
- 18.Streaming tokens to the user as they are generated changes which metric the most?Serving, latency and cost
Sources
Hiring loops change. Every claim above carries a retrieval date so you can judge how current it is.
- [1]Anthropic, Guidance on Candidates' AI Usage · retrieved 2026-08-13
- [2]Anthropic Engineering, Designing AI-resistant technical evaluations · retrieved 2026-08-13
- [3]Anthropic, Careers · retrieved 2026-08-13
Refresh your memory
Free learning paths covering the ground this loop tests, whatever your score. Each one ends with a shareable certificate.
- AIHow LLM apps work
A practical seven-lesson path from raw model calls to a production-ready RAG application. You'll learn how to ground LLMs in your own data, orchestrate the pieces with LangChain, observe what they actually do, and evaluate the result so you can ship with confidence.
7 lessons - AIAdvanced RAG: graphs, hybrid retrieval, and evaluation
Nine lessons that take Retrieval-Augmented Generation past the cosine-search baseline. Chunking, hybrid retrieval with rerankers, query rewriting, GraphRAG and HippoRAG/RAPTOR, agentic multi-hop patterns, and how to actually evaluate the whole thing with RAGAS.
9 lessons - AILLM, agent, and RAG evaluation & monitoring
A compact track on how to know your LLM-powered system is actually working: LLM benchmarks vs. real evals, agent trajectory evaluation, RAG evaluation in production, and the OpenTelemetry GenAI conventions that tie traces across LangSmith, Phoenix, and Datadog LLM.
6 lessons - AIHow Agentic AI systems work
Move beyond single prompts into agents that plan, use tools, and loop. Six lessons covering the agent mindset, the LangGraph framework, the realities of context windows, where humans belong in the loop, the security pitfalls, and how to evaluate non-deterministic systems.
6 lessons - AIFine-Tuning versus RAG: A Decision Framework
They are usually presented as alternatives when they change different things: fine-tuning adjusts behaviour, retrieval supplies knowledge. This cursus makes the choice properly. Why fine-tuning is a poor way to add facts, what it is genuinely good at, and the three questions that settle the matter on their own. Then what building each involves, from LoRA and the training-data problem that stalls most projects to the retrieval pipeline. Then the options the framing hides, long context, prompt caching, agentic retrieval, and how to revisit a decision that has expired.
3 lessons
AI