AI Engineering Interview Questions: Evaluation, Safety & Guardrails

LLM-as-a-judge is a popular evaluation technique, but it has well-documented systematic biases. Studies (e.g., from Zheng et al. 2023 on MT-Bench) show that GPT-4 and similar judges exhibit (1) position bias — preferring the first answer shown, (2) verbosity bias — favouring longer answers regardless of correctness, and (3) self-enhancement bias — rating outputs from models of the same family higher. Calibration drift and sycophancy are real but are properties of the target model, not the judge. The key insight for senior engineers is that using a single LLM judge without positional swapping, length normalization, or multi-judge consensus will silently skew evaluation results.

The FIND_THE_BUG here targets a subtle but critical mistake: the moderate function calls openai.Moderation.create but then checks results[0]['flagged'] using dict-style access on a Pydantic model object returned by the newer OpenAI Python SDK (v1+). The correct attribute access is results.results[0].flagged (dot notation on the model object). Additionally, the guard short-circuits on None return but the caller never checks for None, meaning a flagged message silently disappears without raising an error or returning a meaningful response — but the primary code-level bug is the results[0]['flagged'] dict access which raises a TypeError at runtime with the v1 SDK. The buggy line is line 6.

This question tests deep knowledge of layered guardrail architecture. Input guardrails run before the LLM call to block malicious or off-topic prompts. Output guardrails run after the LLM response to catch hallucinations, PII leakage, or policy violations before delivery to the user. A correct defence-in-depth pipeline therefore needs both layers. Relying only on input guardrails misses prompt-injection-induced unsafe outputs; relying only on output guardrails still allows prompt injection to reach the model, wasting compute and risking data exfiltration. Grounding checks (RAG relevance) are an output-side concern. Rate limiting is an infrastructure concern orthogonal to content safety. A well-designed system applies: (1) input validation → (2) LLM call → (3) output validation, with each stage having independent failure modes.

temperature scales the logits before sampling; at 0 the model becomes (near) greedy — it picks the highest-probability token at each step — which is what you want for deterministic, reproducible extraction. Note the honest caveat: even at temperature 0 you can still see small variation in practice from non-deterministic floating-point reduction order on GPUs, batching, and model updates, so treat it as low variance, not a hard guarantee. Raising temperature (a) does the opposite — it flattens the distribution and increases randomness. Raising top_p (c) widens the nucleus of candidate tokens, again adding variability rather than removing it. frequency_penalty (d) only discourages token repetition; it changes what is generated but does nothing for determinism.

Cosine similarity is the cosine of the angle between two vectors: it is the dot product divided by the product of the magnitudes, so it depends only on direction, not length. That is why it works well for embeddings — two passages about the same topic point the same way regardless of how long each text is. Euclidean distance (a) is a different metric that is sensitive to magnitude (though on length-normalized vectors the two ranking orders coincide). "Shared dimensions" (c) is not a real similarity measure for dense embeddings, whose dimensions are not independently interpretable. Token overlap (d) describes lexical/keyword matching (e.g. BM25), which is exactly what dense embeddings are meant to go beyond.

RAG is the right default when knowledge is large, changes frequently, and answers need attribution: you index the documents, retrieve the relevant chunks per query, and the model answers from them — so updates are just re-indexing, and you can cite the chunks you retrieved. Nightly fine-tuning (a) is slow, expensive, hard to attribute (the model can't reliably cite which training example produced an answer), and it bakes facts into weights where they go stale between runs. Stuffing the whole KB into the system prompt (c) blows the context window and cost, and degrades quality as irrelevant text dilutes the signal. A one-time LoRA on a snapshot (d) is immediately stale for daily-changing data and still can't cite sources. Fine-tuning earns its place for behavior/format/tone, not volatile facts.

Provider structured-output features (e.g. JSON Schema-constrained decoding) restrict the token sampler at each step to tokens that keep the output valid against the schema, so the result is guaranteed-parseable and schema-conformant by construction. A free-text instruction (a) is best-effort: the model can still emit prose, trailing commas, or extra keys. Temperature 0 (c) only reduces variability — a deterministic output can be deterministically malformed. Retry-on-error (d) is a reasonable fallback and improves reliability, but it adds latency and cost and still has no hard guarantee on any single attempt; constrained decoding removes the failure mode at the source.

Function/tool calling is a protocol, not remote execution: given your tool definitions (name, description, JSON-Schema parameters), the model decides whether to call one and emits a structured tool-call with arguments. Your code executes the tool, then sends the result back as a tool/function message so the model can use it in its next turn. The provider does not run your code (a) — it has no access to your database or APIs. The model returns structured arguments, not executable code to eval (c); eval-ing model output would be a serious injection risk. And it is a concrete, parseable call, not a vague wish (d).

This is prompt injection: an LLM processes its entire context as one token stream and has no robust, built-in boundary between trusted instructions and untrusted data, so adversarial text embedded in fetched content can hijack behavior. The defense is architectural — keep untrusted content clearly delimited, apply least-privilege to tools, and require human approval or hard policy checks for dangerous actions like sending email or exfiltrating secrets — not a single magic setting. Temperature (a) controls randomness, not whether instructions are followed. Lengthening the system prompt (c) does not create a real trust boundary; a sufficiently crafted injection can still override it. Embedding dimensionality (d) is about retrieval quality and is unrelated to the model obeying injected commands.

Prompt caching stores the processed key/value state for a stable prefix (your unchanging system prompt + policy doc); on later requests the model reuses it instead of re-processing those tokens, which lowers input cost and time-to-first-token while leaving outputs identical — ideal when a large prefix is constant and only the tail varies. Changing temperature (b) affects randomness, not cost. Retrieving the system prompt via vector search (c) adds machinery and a retrieval step to fetch text you already have verbatim — pure overhead. Raising max_tokens (d) only sets an upper bound on output length; it does not reduce the cost of re-reading the same input every call and can increase cost if it lets responses grow.

For open-ended generation, LLM-as-judge over a fixed, versioned eval set scales and correlates reasonably with human judgment when you give the judge a clear rubric and, ideally, a reference answer. The essential caveat is that the judge is itself a fallible model — it shows known biases (position bias, verbosity/length bias, self-preference) — so you must validate it against a sample of human labels and re-check when you change the judge model. Exact match (a) fails by design for free-form text where many wordings are equally correct. BLEU against a single reference (c) was built for machine translation and penalizes valid paraphrases; its weakness is poor correlation with quality on summaries, not speed. Reading every output manually (d) does not scale to weekly iteration; spot-checking belongs alongside an automated eval, not as the primary gate.

Retrieval quality is about getting the right chunks into context. A reranker (a) reorders the initial candidate set with a more expensive, more accurate cross-encoder so the best passages float to the top. Chunking strategy (b) directly affects whether a chunk contains a complete, embeddable idea rather than a fragment. Hybrid retrieval (c) catches cases where exact terms/IDs matter that dense vectors miss, and vice versa. A better-matched embedding model (e) improves the similarity signal at the source. Raising max_tokens (d) only changes how long the answer may be — it does nothing about which documents were retrieved, so it cannot fix retrieving the wrong material.

Streaming (a) doesn't change total compute but dramatically improves perceived speed by showing the first tokens immediately. Model routing / cascading (b) sends the bulk of easy traffic to a cheaper model, cutting average cost and latency while preserving quality on the hard tail. Caching (c) avoids paying for work you've already done. Trimming context and bounding output length (e) reduces both input and output tokens, which is where the bill and the time go. Indiscriminately padding every prompt with more few-shot examples (d) does the opposite — it inflates input tokens (cost and latency) on every call, and beyond a point adds no accuracy, so it is a regression, not an optimization.

Hallucination drops when the model is anchored to real evidence: retrieval-grounding with an instruction to answer only from context (a) constrains it to supported claims. An explicit "I don't know" escape hatch (b) gives the model a correct option other than fabricating, which it otherwise tends to avoid. Citation + verification (c) turns grounding into something you can check, catching claims that aren't actually supported. Raising temperature (d) increases randomness and makes confident fabrication more likely, not less. A bare instruction to "never hallucinate" (e) is wishful — the model has no reliable internal signal of its own factuality, so an unenforceable command doesn't change behavior in a measurable way; grounding and verification do.

MCP is an open, client-server protocol that gives a uniform way to expose tools, resources, and prompts to LLM hosts (a), and its central payoff is write-once/reuse-everywhere interoperability across compatible clients (b). It does not remove the security burden: a server runs with whatever access you give it and the data it returns enters the model's context, so an untrusted or compromised server is a genuine injection/exfiltration risk (c) — least privilege and review still apply. MCP does not replace function calling (d); it is the transport/standard through which tool definitions and calls flow — the model still decides which tool to invoke. And it offers no guarantee against malicious server output misleading the model (e); content returned over MCP is untrusted data like any other.