AI Engineering Interview Questions: LLM Foundations & Model Selection

Rotary Position Embeddings (RoPE) encode position by rotating query and key vectors in pairs within the attention computation. The key property is that the dot-product between a query at position m and a key at position n depends only on the relative offset (m − n), not their absolute positions — this emerges naturally from the rotation mathematics. This is why RoPE-based models generalize better to sequence lengths beyond their training context: the relative distances are inherently represented. ALiBi (Attention with Linear Biases) achieves a similar effect differently — it subtracts a linear position bias directly on attention logits rather than modifying Q/K vectors. The critical distinction is that RoPE operates in the Q/K embedding space, while ALiBi operates on the attention score matrix post-QK dot-product. Extended context methods like YaRN and LongRoPE scale the RoPE frequency base (theta) — NOT the embedding dimension — to accommodate longer sequences without full fine-tuning.

Speculative decoding uses a small, fast 'draft' model to propose multiple candidate tokens in parallel, then the large 'verifier' model checks them in a single forward pass. The acceptance criterion is designed so that the output distribution is identical to sampling from the verifier alone — this is the key correctness guarantee. The speedup comes from the fact that verifying K tokens costs only slightly more than verifying 1 (batch dimension), while the draft model is cheap. Critically, speculative decoding does NOT change the output distribution of the target model; it only accelerates sampling. The acceptance rate (α) depends on how well the draft model's distribution matches the verifier's. If the draft model is too mismatched (low α), the overhead of running both models exceeds the gain, negating the speedup. Self-speculative decoding (e.g., Medusa heads, EAGLE) avoids the need for a separate draft model by using auxiliary prediction heads on the base model itself.

Perplexity measures how well a language model predicts a sample of text — lower perplexity means the model assigns higher probability to the test corpus and is therefore a better fit for that domain. BLEU is a translation/generation quality metric, FLOPS measures compute, and top-k accuracy is a classification metric. When selecting a model for a domain-specific language task, perplexity on held-out domain text is the most directly relevant intrinsic metric for assessing language modeling quality before any task-specific fine-tuning or evaluation.

temperature scales the logits before sampling; at 0 the model becomes (near) greedy — it picks the highest-probability token at each step — which is what you want for deterministic, reproducible extraction. Note the honest caveat: even at temperature 0 you can still see small variation in practice from non-deterministic floating-point reduction order on GPUs, batching, and model updates, so treat it as low variance, not a hard guarantee. Raising temperature (a) does the opposite — it flattens the distribution and increases randomness. Raising top_p (c) widens the nucleus of candidate tokens, again adding variability rather than removing it. frequency_penalty (d) only discourages token repetition; it changes what is generated but does nothing for determinism.

Cosine similarity is the cosine of the angle between two vectors: it is the dot product divided by the product of the magnitudes, so it depends only on direction, not length. That is why it works well for embeddings — two passages about the same topic point the same way regardless of how long each text is. Euclidean distance (a) is a different metric that is sensitive to magnitude (though on length-normalized vectors the two ranking orders coincide). "Shared dimensions" (c) is not a real similarity measure for dense embeddings, whose dimensions are not independently interpretable. Token overlap (d) describes lexical/keyword matching (e.g. BM25), which is exactly what dense embeddings are meant to go beyond.

RAG is the right default when knowledge is large, changes frequently, and answers need attribution: you index the documents, retrieve the relevant chunks per query, and the model answers from them — so updates are just re-indexing, and you can cite the chunks you retrieved. Nightly fine-tuning (a) is slow, expensive, hard to attribute (the model can't reliably cite which training example produced an answer), and it bakes facts into weights where they go stale between runs. Stuffing the whole KB into the system prompt (c) blows the context window and cost, and degrades quality as irrelevant text dilutes the signal. A one-time LoRA on a snapshot (d) is immediately stale for daily-changing data and still can't cite sources. Fine-tuning earns its place for behavior/format/tone, not volatile facts.

Provider structured-output features (e.g. JSON Schema-constrained decoding) restrict the token sampler at each step to tokens that keep the output valid against the schema, so the result is guaranteed-parseable and schema-conformant by construction. A free-text instruction (a) is best-effort: the model can still emit prose, trailing commas, or extra keys. Temperature 0 (c) only reduces variability — a deterministic output can be deterministically malformed. Retry-on-error (d) is a reasonable fallback and improves reliability, but it adds latency and cost and still has no hard guarantee on any single attempt; constrained decoding removes the failure mode at the source.

Function/tool calling is a protocol, not remote execution: given your tool definitions (name, description, JSON-Schema parameters), the model decides whether to call one and emits a structured tool-call with arguments. Your code executes the tool, then sends the result back as a tool/function message so the model can use it in its next turn. The provider does not run your code (a) — it has no access to your database or APIs. The model returns structured arguments, not executable code to eval (c); eval-ing model output would be a serious injection risk. And it is a concrete, parseable call, not a vague wish (d).

This is prompt injection: an LLM processes its entire context as one token stream and has no robust, built-in boundary between trusted instructions and untrusted data, so adversarial text embedded in fetched content can hijack behavior. The defense is architectural — keep untrusted content clearly delimited, apply least-privilege to tools, and require human approval or hard policy checks for dangerous actions like sending email or exfiltrating secrets — not a single magic setting. Temperature (a) controls randomness, not whether instructions are followed. Lengthening the system prompt (c) does not create a real trust boundary; a sufficiently crafted injection can still override it. Embedding dimensionality (d) is about retrieval quality and is unrelated to the model obeying injected commands.

Retrieval quality is about getting the right chunks into context. A reranker (a) reorders the initial candidate set with a more expensive, more accurate cross-encoder so the best passages float to the top. Chunking strategy (b) directly affects whether a chunk contains a complete, embeddable idea rather than a fragment. Hybrid retrieval (c) catches cases where exact terms/IDs matter that dense vectors miss, and vice versa. A better-matched embedding model (e) improves the similarity signal at the source. Raising max_tokens (d) only changes how long the answer may be — it does nothing about which documents were retrieved, so it cannot fix retrieving the wrong material.

Streaming (a) doesn't change total compute but dramatically improves perceived speed by showing the first tokens immediately. Model routing / cascading (b) sends the bulk of easy traffic to a cheaper model, cutting average cost and latency while preserving quality on the hard tail. Caching (c) avoids paying for work you've already done. Trimming context and bounding output length (e) reduces both input and output tokens, which is where the bill and the time go. Indiscriminately padding every prompt with more few-shot examples (d) does the opposite — it inflates input tokens (cost and latency) on every call, and beyond a point adds no accuracy, so it is a regression, not an optimization.

Hallucination drops when the model is anchored to real evidence: retrieval-grounding with an instruction to answer only from context (a) constrains it to supported claims. An explicit "I don't know" escape hatch (b) gives the model a correct option other than fabricating, which it otherwise tends to avoid. Citation + verification (c) turns grounding into something you can check, catching claims that aren't actually supported. Raising temperature (d) increases randomness and makes confident fabrication more likely, not less. A bare instruction to "never hallucinate" (e) is wishful — the model has no reliable internal signal of its own factuality, so an unenforceable command doesn't change behavior in a measurable way; grounding and verification do.

MCP is an open, client-server protocol that gives a uniform way to expose tools, resources, and prompts to LLM hosts (a), and its central payoff is write-once/reuse-everywhere interoperability across compatible clients (b). It does not remove the security burden: a server runs with whatever access you give it and the data it returns enters the model's context, so an untrusted or compromised server is a genuine injection/exfiltration risk (c) — least privilege and review still apply. MCP does not replace function calling (d); it is the transport/standard through which tool definitions and calls flow — the model still decides which tool to invoke. And it offers no guarantee against malicious server output misleading the model (e); content returned over MCP is untrusted data like any other.

The context window bounds the total tokens the model attends to: system prompt + retrieved context + conversation history + the tokens it generates. If you stuff the input close to the limit, you starve the output and risk truncated completions. Budget explicitly — reserve headroom for max_tokens of output, and remember that long outputs cost both latency and window space.

Prompt caching keys on a prefix of the request, so a long, unchanging prefix (system instructions, few-shot examples, a fixed knowledge block) can be cached once and reused — cache reads are billed at a steep discount and skip recomputation, lowering latency. The catch: ordering matters. Anything that varies must come after the stable prefix, otherwise it breaks the cacheable prefix and you pay full price every call. Structure prompts stable-first, variable-last to maximize hits.