LLM Engineer Interview Questions: Top 40 (2026)

This is a 40-question bank for the 2026 LLM engineer loop, grouped into the five clusters that cover almost every interview: transformer internals, fine-tuning, inference optimization, serving and cost, and evals. Each question has a tight answer and the exact signal the interviewer is grading.

LLM engineer postings grew sharply through 2025 as companies moved from "bolt an API on" to "run our own serving stack." Total compensation is reported in the range of roughly $150K to $320K depending on seniority and city per Levels.fyi. The 2026 LLM engineer interview goes deeper than a generic AI engineer list: it tests attention math, KV cache memory, quantization tradeoffs, tokens-per-second throughput, and cost-per-million-token math you may have to compute live. This guide gives you all 40 questions with model answers and grading signals.

These three titles overlap, and interviewers will probe whether you know the difference. An LLM engineer owns the model and the serving stack. You care about attention internals, KV cache layout, quantization, batching schedulers, and tokens-per-second on a specific GPU. The job is making a model run fast, cheap, and correct at scale.

An AI engineer builds products on top of foundation models: RAG pipelines, agent loops, tool wiring, prompt design. They treat the model as an API. If you are prepping that side too, the AI engineer interview questions cover it, and the RAG interview questions go deep on retrieval.

An ML engineer owns training pipelines, feature stores, and classical model deployment across many model types. The machine learning interview questions cover that ground. This page stakes out the LLM-engineer middle: model internals plus inference serving plus token-level evals. That is why it goes deep on KV cache, quantization, and serving math the other lists skip.

Most 2026 LLM engineer loops run four to five rounds across five topic clusters: transformers and attention, fine-tuning vs RAG vs prompting, inference optimization, serving and cost, and evals and guardrails. A recruiter screen is followed by a technical phone screen, then an onsite (often virtual) of three to four rounds.

Where candidates lose offers: they can recite the attention formula but cannot reason about KV cache memory in gigabytes. They name "quantization" but cannot compare INT4 to FP8 on accuracy. They have never computed a cost-per-million-token figure. Senior loops expect you to do that math on the whiteboard without notes.

A growing share of loops also include a from-scratch coding round. You implement attention or a KV cache in PyTorch, or sketch a batching scheduler. It is not LeetCode. It tests whether you understand the mechanics well enough to build them, not just describe them. Budget prep time for both the conceptual Q&A and the live coding.

These nine test whether you understand the machine you are serving. Fumble more than two and the inference rounds get harder.

Answer: Each token projects into query, key, and value vectors. Attention scores are the softmax of QK^T scaled by 1/√d_k, and you multiply those weights by V to get a context-aware representation per token. The full form is softmax(QK^T / √d_k)V. The √d_k scaling keeps the dot products from growing with dimension and saturating the softmax.

What they want: Whether you can derive it cold, including why the scaling factor exists.

Answer: Single-head attention computes one set of Q/K/V projections, so it can only attend in one representational subspace. Multi-head splits the model dimension across h heads, each with its own projections, then concatenates and projects the results. This lets the model attend to syntax, coreference, and position in parallel subspaces.

What they want: Whether you know why heads exist instead of one big attention.

Answer: Grouped-query attention shares a small number of key/value heads across many query heads, sitting between full multi-head attention and multi-query attention (one KV head). It shrinks the KV cache by the grouping factor with almost no quality loss. Models moved to GQA because the KV cache, not compute, is the memory bottleneck at long context and large batch.

What they want: Whether you connect an architecture choice to a serving constraint.

Answer: Rotary position embeddings rotate the query and key vectors by an angle proportional to position, so the dot product between two tokens depends on their relative distance. It encodes relative position directly into attention, extrapolates to longer sequences better than absolute learned embeddings, and supports context-length extension tricks like NTK scaling and YaRN.

What they want: Whether you understand position encoding beyond "we add a vector."

Answer: Every token attends to every other token, so the attention matrix is n×n in both compute and memory for sequence length n. At long context the quadratic term dominates prefill cost and can exhaust memory. This is why FlashAttention tiles the computation to avoid materializing the full matrix, and why long-context serving is expensive even before the KV cache.

What they want: Whether you see the scaling wall that defines long-context serving.

Answer: Models bill per token, so a language or format that tokenizes inefficiently (Thai, Korean, dense code, long digit strings) can cost three to five times more than English for the same content. Tokenization also affects context-window math, where the model splits words for streaming, and arithmetic behavior, since numbers get chopped into odd pieces.

What they want: Whether you have done real unit economics, not just theory.

Answer: The context window is the maximum tokens the model can attend to in one forward pass, commonly 128K to 2M for frontier models in 2026. The cost is not free: KV cache memory grows linearly with context length, so a 1M-token context can need tens of gigabytes of cache. Exceed the window and you either truncate or get a 400 error, so your history and retrieval strategy must fit the budget.

What they want: Whether you tie context length back to memory cost, not just capability.

Answer: An embedding maps text to a dense vector where semantic similarity becomes geometric proximity. Token embeddings are per-token lookups inside the model; sentence or document embeddings pool a sequence into one vector for retrieval. For retrieval you pick an embedding model by MTEB performance in your domain, dimension count (affecting storage two to four times), and whether you need multilingual or code variants.

What they want: Whether you separate the model's internal embeddings from retrieval embeddings.

Answer: The token ID is embedded, run through each transformer block where attention reads the KV cache and the MLP transforms the residual stream, then the final hidden state hits the unembedding matrix to produce logits over the vocabulary. You apply temperature and top-p, sample one token, append its K and V to the cache, and repeat. Prefill processes the whole prompt at once; decode is one token at a time.

What they want: Whether you can trace the full forward pass, which sets up every inference question.

This cluster tests whether you reach for the right tool and know the cost of each. The deep learning interview questions cover the gradient mechanics underneath these.

Answer: Prompt first for behavior you can specify in instructions and few-shot examples. Reach for RAG when the model needs fresh, large, or auditable knowledge, since it stays frozen and you update the index. Fine-tune when you need a consistent style, a narrow format, or a latency win from a smaller specialized model, and only when prompting and RAG have hit their ceiling.

What they want: Whether you know most "we need to fine-tune" requests are actually RAG or prompting problems.

Answer: LoRA freezes the base weights and learns two low-rank matrices A and B whose product is added to a weight matrix, so you train rank×dimension parameters instead of the full matrix. That cuts trainable parameters by 100 to 1000 times, fits on a single GPU, and produces small swappable adapters. At inference you can merge the adapter into the weights or keep it separate to hot-swap behaviors.

What they want: Whether you understand parameter-efficient tuning mechanically, not as a buzzword.

Answer: QLoRA quantizes the frozen base model to 4-bit (NF4) and trains LoRA adapters on top, so the giant base weights sit in 4-bit while gradients flow only through the small adapters. The QLoRA paper showed this fine-tunes a 65B model on a single 48GB GPU while matching 16-bit performance. The tradeoffs are slightly slower training from dequantization and minor quality loss versus full-precision LoRA.

What they want: Whether you can fine-tune large models on constrained hardware.

Answer: Full fine-tuning updates all weights, needs many times the model size in memory for optimizer states, and risks overwriting general capabilities. PEFT (LoRA, adapters, prefix tuning) updates a tiny fraction, is cheaper and safer, and is the default in 2026. Full fine-tuning wins only when you have a large high-quality dataset and need to shift the model's core behavior, not just steer it.

What they want: Whether you default to PEFT and know the rare case for full tuning.

Answer: Instruction tuning is supervised fine-tuning on (instruction, response) pairs so a base completion model learns to follow directions instead of just continuing text. It is what turns a raw pretrained model into something usable as an assistant. It comes before preference optimization in the post-training pipeline and sets the model's default helpfulness and format.

What they want: Whether you know the post-training pipeline, not just pretraining.

Answer: PPO-based RLHF trains a separate reward model, then uses reinforcement learning to optimize the policy against it, which is powerful but unstable and infrastructure-heavy. DPO skips the reward model and RL loop, optimizing a classification-style loss directly on preference pairs. By 2026 DPO and its variants are common because they get most of the alignment benefit with far simpler, more stable training.

What they want: Whether you track the shift from PPO to DPO and know why it happened.

Answer: Catastrophic forgetting is when fine-tuning on a narrow dataset erodes the model's general capabilities because the weights move to fit the new task. Mitigations include mixing in general data during fine-tuning, using PEFT so most weights stay frozen, lowering the learning rate, and adding a KL penalty against the base model. Always eval on a general benchmark, not just your task set.

What they want: Whether you anticipate the regression that full fine-tuning causes.

Answer: For style and format with LoRA, a few hundred to a few thousand high-quality examples often suffice. For new behavior or domain depth, expect tens of thousands. Quality and diversity beat raw count: a thousand clean, consistent examples outperform ten thousand noisy ones, and a few mislabeled examples can poison a small set. Always hold out an eval split before you start.

What they want: Whether you size data realistically instead of saying "more is better."

This is the heart of the LLM engineer role and where generic AI engineer lists go silent. Expect real numbers.

Answer: The KV cache stores the key and value tensors for every past token so each new token does not recompute attention over the whole prefix, making decode linear instead of quadratic. Its size is roughly 2 (K and V) × num_layers × num_kv_heads × head_dim × sequence_length × batch_size × bytes_per_element. For a large model at long context and high batch, that is tens of gigabytes, often rivaling the weights themselves.

What they want: Whether you can compute KV memory in gigabytes on the spot.

Answer: Prefill processes the entire prompt in one parallel forward pass, so it is compute-bound and fast per token. Decode generates one token at a time reading the growing KV cache, so it is memory-bandwidth-bound and slow per token. The split matters because they have different bottlenecks: prefill wants compute (FLOPs), decode wants bandwidth, and serving systems schedule them differently.

What they want: Whether you know the two phases have opposite hardware bottlenecks.

Answer: GPTQ and AWQ are post-training weight-quantization methods to INT4; AWQ protects salient weight channels by activation scale and often retains a touch more accuracy. INT8 weight quantization typically retains roughly 99% of quality, INT4 around 95 to 99% with good methods, and FP8 retains near-full accuracy while being hardware-native on H100-class GPUs. FP8 is the 2026 default for high-throughput serving; INT4 wins when memory is the hard constraint.

What they want: Whether you can compare methods with accuracy-retention numbers, not just say "we quantize."

Answer: Weight-only quantization (GPTQ, AWQ) compresses the stored weights and dequantizes during compute, which shrinks memory and helps the bandwidth-bound decode phase. Activation quantization also quantizes the running activations so the matrix multiplies run in low precision (INT8 or FP8), which speeds the compute-bound prefill but is more sensitive to outliers. FP8 with per-tensor or per-channel scaling is the common way to do both safely.

What they want: Whether you know which quantization helps which phase.

Answer: Static batching waits for the whole batch to finish, so a short request is stuck behind a long one and the GPU idles. Continuous (in-flight) batching swaps finished sequences out and new ones in at the token level, keeping the GPU saturated. It can multiply throughput several times over static batching with no model change, which is why every modern serving stack uses it.

What they want: Whether you know the single highest-leverage serving optimization.

Answer: Naive KV cache allocation reserves contiguous memory for the max sequence length, wasting most of it to fragmentation. PagedAttention stores the KV cache in fixed-size non-contiguous blocks like OS virtual memory paging, cutting waste to near zero and enabling prefix sharing across requests. The vLLM PagedAttention paper reports 2 to 4 times higher throughput at the same latency over prior serving systems, and it pairs this with continuous batching to push throughput far above naive serving.

What they want: Whether you understand the memory-management problem vLLM actually fixed.

Answer: A small fast draft model proposes several tokens, then the large target model verifies them all in a single parallel forward pass, accepting the longest correct prefix. Because verification is parallel and the draft is cheap, you get roughly 2 to 3 times faster decode with identical output distribution. The win depends on draft acceptance rate, so the draft must be well aligned with the target.

What they want: Whether you know a lossless latency optimization and its 2 to 3 times ceiling.

Answer: FlashAttention computes attention in tiles in fast SRAM without materializing the full n×n matrix in HBM, turning attention from memory-bound to compute-efficient and cutting memory from quadratic to linear. FlashAttention-3 targets Hopper GPUs with FP8 support and better overlap of compute and data movement, reaching up to 75% utilization on H100 in FP16 (1.5 to 2 times faster than FlashAttention-2). It speeds both training and prefill at long context.

What they want: Whether you know the kernel that made long-context training and serving feasible.

Answer: A mixture-of-experts model routes each token to a small number of experts (say 2 of 8) via a gating network, so only a fraction of parameters activate per token. You get the quality of a huge model at the compute cost of a small one, but you must hold all experts in memory, and uneven routing causes load imbalance across GPUs. Expert parallelism and capacity factors are the levers to keep it balanced.

What they want: Whether you know MoE saves compute but not memory, and why routing is tricky to serve.

Answer: Decode reads all weights and the KV cache per token, so the fixes attack bytes moved: quantize weights to INT4 or FP8 to halve or quarter the bytes, use GQA to shrink the KV cache, batch more requests so the weight read is amortized across sequences, and apply speculative decoding so you verify several tokens per forward pass. Bigger batches and smaller precision are the highest-leverage moves.

What they want: Whether you map the bandwidth bottleneck to the right set of fixes.

Senior loops expect you to compute these live. Practice the arithmetic until it is automatic.

Answer: Memory has three buckets: weights, KV cache, and activation overhead. Weights are roughly num_parameters × bytes_per_param, so a 70B model is about 140GB in FP16 or 35GB in INT4. The KV cache scales with batch size and context length and can reach tens of gigabytes. Activations and framework overhead add a buffer, so on an 80GB GPU a 70B model needs quantization plus tensor parallelism, while a 13B model fits comfortably with room for a large batch.

What they want: Whether you can size a deployment to real GPU memory.

Answer: Throughput is total output tokens divided by wall-clock time, measured at the batch level since one request alone underuses the GPU. You raise it with continuous batching, larger batch sizes until you hit a latency or memory ceiling, quantization, tensor parallelism, and a serving stack like vLLM or TensorRT-LLM. Always separate single-request latency from aggregate throughput; optimizing one can hurt the other.

What they want: Whether you distinguish per-request latency from system throughput.

Answer: Break the budget into time-to-first-token (driven by prefill and queue wait) and inter-token latency (driven by decode speed), then track p95, not just the mean, since tail latency is what users feel. Stream tokens so TTFT can be 200 to 400ms even if total is longer, cap batch size to bound queue delay, and route long prompts to a separate lane. If p95 misses target, attack the largest bucket in the breakdown.

What they want: Whether you reason about tail latency and streaming, not averages.

Answer: Take the GPU hourly cost (say $2/hour for an H100 on-demand), divide by throughput in tokens per second to get cost per token, then scale to a million. At 2,000 tokens/sec that GPU produces 7.2M tokens/hour, so a million tokens costs about $0.28. Higher batch and quantization raise throughput and lower this number; idle GPUs and low batch utilization wreck it. Compare that figure against the API price to decide build vs buy.

What they want: Whether you can derive cost per million tokens live from GPU price and throughput.

Answer: Streaming sends tokens as they are generated over server-sent events instead of waiting for the full response, so perceived latency drops to time-to-first-token even when total generation is slow. It changes the architecture: you need an SSE or WebSocket transport, partial output handling, and mid-stream cancellation to stop billing when a user navigates away. It does not speed total generation; it just front-loads the experience.

What they want: Whether you know streaming is a UX and cost lever, not a speedup.

Answer: Loading a large model into GPU memory takes tens of seconds to minutes, so naive autoscaling gives brutal cold starts. You keep a warm pool of minimum replicas, scale on queue depth or in-flight requests rather than CPU, use fast model loading and snapshotting, and pre-warm before known traffic spikes. For spiky low-volume workloads, a serverless or API provider often beats self-hosting on cost.

What they want: Whether you have felt GPU cold starts and designed around them.

Answer: A hosted API wins on time to market, zero infra, and elastic scale, and is cheaper at low or spiky volume. Self-hosting wins above a high steady volume (often millions of tokens per day), under data-residency or privacy constraints, when you need custom fine-tunes or quantization, or for latency-sensitive co-located inference. Run both in shadow mode on real traffic and let the cost-per-million-token and p95 numbers decide.

What they want: Whether you make the call with arithmetic instead of preference.

Eval methodology is the new system design. Expect at least one full round here.

Answer: A golden set is a curated collection of representative inputs with known-good outputs or graded rubrics, drawn from real production traffic, edge cases, and known failure modes. You version it, keep it small enough to run on every change but broad enough to catch regressions, and grow it by adding every new bug you find as a test case. It is the foundation every other eval builds on.

What they want: Whether you treat evals as versioned engineering artifacts.

Answer: LLM-as-judge uses a strong model to grade another model's output against a rubric, which scales far past human grading. Its failure modes are real: position bias (favoring the first answer), verbosity bias (preferring longer responses), self-preference for its own model family, and inconsistency across runs. You mitigate with a tight rubric, randomized order, few-shot calibration examples, and human spot-checks on a sample.

What they want: Whether you trust LLM-as-judge but know exactly where it lies.

Answer: Faithfulness checks whether every claim in the answer is supported by the retrieved context, and groundedness checks the answer does not invent facts beyond it. You measure both by decomposing the answer into atomic claims and verifying each against the source, often with an LLM-judge or an NLI model. Low scores point to either a retrieval gap or a model ignoring its context. The prompt engineer interview questions cover the prompt-side levers for this.

What they want: Whether you can quantify hallucination instead of eyeballing it.

Answer: Run the golden set in CI on every prompt, model, or config change and gate the deploy on eval scores the way you gate on unit tests. Track per-category metrics so a fix for one case does not silently break another, and shadow-deploy new versions on a slice of live traffic before full rollout. The point is to make eval scores a deploy-blocking signal, not a quarterly report.

What they want: Whether evals are wired into CI/CD or just run ad hoc.

Answer: Ground the model with RAG and enforce citations so unsupported claims are visible, lower temperature for factual tasks, validate structured outputs against a schema, and add an LLM-judge or NLI check that flags answers not grounded in the source. For high-stakes paths, require the model to abstain or escalate when confidence or retrieval quality is low. Treat it as a set of engineering knobs, not a model defect.

What they want: Whether you see hallucination as tunable, not fatal.

Answer: Separate trusted instructions from untrusted input, never let retrieved or user content silently become system instructions, and scope tool permissions so a hijacked prompt cannot do real damage. On the output side, validate against a schema, scan for leaked secrets or PII, and run a moderation check before anything reaches a user or executes. Defense in depth, since no single filter is reliable.

What they want: Whether you treat untrusted input as an attack surface, not just a string.

Many 2026 loops add a live coding round. It is not LeetCode. It tests whether you understand the mechanics well enough to build them.

The most common ask is implementing attention from scratch in PyTorch: Q/K/V projections, scaled dot-product, softmax, the output projection, then extending to multi-head. The next tier is a KV cache: maintain per-layer key and value tensors across decode steps and show that the second token does not recompute the first. You may also be asked to write a minimal RAG loop (embed, retrieve top-K, stuff context, call the model) or sketch a continuous-batching scheduler that swaps finished sequences out and new ones in at the token level.

Graders look for correct tensor shapes, the √d_k scaling, numerically stable softmax, and clean handling of the batch and sequence dimensions. They care more about whether you reason out loud about memory and shapes than whether you finish. Saying "this is O(n²) in sequence length, here is where the KV cache fixes it" earns more than silent correct code.

In the loops I have watched, the candidates who pass this round are almost never the ones who type the fastest. They are the ones who narrate the shapes before writing a line: "Q is (batch, heads, seq, head_dim), K transposed gives me scores of (batch, heads, seq, seq), so this is where my memory blows up at long context." That one sentence tells the grader you understand the machine, not just the API, and it buys you slack if your code has a bug. The candidates who freeze are the ones who memorized softmax(QK^T / √d_k)V as a string and never traced a single tensor through it.

Study in dependency order. Lock transformer internals and one decode trace first, because every inference question builds on them. Then fine-tuning (LoRA, QLoRA, DPO), then the inference cluster (KV cache, quantization, batching, speculative decoding), then serving and cost math, then evals. Do not skip the math: practice KV cache sizing and cost-per-million-token until you can do them on a whiteboard without notes.

Drill under time pressure. Set a two-minute timer per question and speak the answer out loud, since reading is not the same as recall. Build one thing with your hands: serve a small model with vLLM, watch tokens-per-second move as you change batch size and quantization. That single exercise makes the serving cluster concrete. If you are also interviewing for agent-heavy roles, the agentic AI interview questions pair well with this set. Then run timed mock loops with feedback until two minutes per question feels easy.

Is LLM engineer harder than AI engineer? It is more systems-heavy. AI engineer leans on RAG and agent wiring; LLM engineer adds attention internals, GPU memory math, quantization, and serving throughput. If you like low-level performance work, it is the better fit. If you prefer product wiring, AI engineer suits you more.

Do you need a PhD? No. You need to understand transformer internals, inference mechanics, and serving cost well enough to reason live. Most LLM engineers come from strong software or ML-systems backgrounds, not research. A PhD helps for frontier-lab research roles, not for serving and applied work.

What salary should I expect in 2026? Total compensation is reported in the range of roughly $150K to $320K depending on seniority and city per Levels.fyi, with senior roles at frontier labs and AI scaleups pushing higher with equity. The systems and serving skill set commands a premium because fewer engineers have it.

How long should I prep? Two to four weeks of focused work clears most loops if you already code. Spend the first week on internals, the middle on the inference and serving clusters and their math, and the last on mock interviews and the from-scratch coding round.

Do you need to train models from scratch? Rarely. The role centers on fine-tuning, serving, and optimizing existing models, not pretraining. You should understand training mechanics enough to fine-tune and to reason about tradeoffs, but you will almost never train a base model in this job.

The 40 questions above are the floor. The candidates who get offers in 2026 can compute KV cache memory and cost-per-million-token on a whiteboard, compare quantization methods with real accuracy numbers, and implement attention from scratch without freezing. That fluency comes from reps, not reading.

If you want timed reps in the same environment as a real loop, Interview Coder runs mock LLM engineer sessions with live feedback and full session logs so you can see where you actually improved. Full disclosure: this guide is published by Interview Coder, its own product. It is a desktop app with 20+ stealth features and 100K+ users; its coding answers run on Claude Sonnet 4.6, Anthropic's latest Sonnet. Plans are Free at $0, Monthly Pro at $299, and Lifetime Pro at $799 one-time. The grind is still the grind, but you can stop guessing whether your prep is working.