AI Engineering Interview Questions: Agents & Agent Loops

The loop is: present the model with the tools it may call, the model decides to emit a tool call with structured arguments, your code executes that tool (the model never runs it), you return the result as a tool/function message appended to the conversation, and the model then synthesizes the final answer from it. For multi-step tasks the middle three steps repeat until the model stops requesting tools.

An agent loop (also called a ReAct loop or think-act loop) follows a repeated cycle: the agent observes the current state/context, reasons or thinks about what to do, takes an action (e.g., calls a tool), receives an observation back, and repeats until a stopping condition is met. This is the fundamental pattern behind virtually all LLM-based agent frameworks.

In an agent loop the LLM does NOT execute code or call APIs directly — that is the job of the surrounding runtime/executor. The LLM's role is limited to reasoning and deciding which action to take and what arguments to pass. The runtime then actually performs the action and returns the result (observation) to the LLM.

temperature scales the logits before sampling; at 0 the model becomes (near) greedy — it picks the highest-probability token at each step — which is what you want for deterministic, reproducible extraction. Note the honest caveat: even at temperature 0 you can still see small variation in practice from non-deterministic floating-point reduction order on GPUs, batching, and model updates, so treat it as low variance, not a hard guarantee. Raising temperature (a) does the opposite — it flattens the distribution and increases randomness. Raising top_p (c) widens the nucleus of candidate tokens, again adding variability rather than removing it. frequency_penalty (d) only discourages token repetition; it changes what is generated but does nothing for determinism.

Cosine similarity is the cosine of the angle between two vectors: it is the dot product divided by the product of the magnitudes, so it depends only on direction, not length. That is why it works well for embeddings — two passages about the same topic point the same way regardless of how long each text is. Euclidean distance (a) is a different metric that is sensitive to magnitude (though on length-normalized vectors the two ranking orders coincide). "Shared dimensions" (c) is not a real similarity measure for dense embeddings, whose dimensions are not independently interpretable. Token overlap (d) describes lexical/keyword matching (e.g. BM25), which is exactly what dense embeddings are meant to go beyond.

RAG is the right default when knowledge is large, changes frequently, and answers need attribution: you index the documents, retrieve the relevant chunks per query, and the model answers from them — so updates are just re-indexing, and you can cite the chunks you retrieved. Nightly fine-tuning (a) is slow, expensive, hard to attribute (the model can't reliably cite which training example produced an answer), and it bakes facts into weights where they go stale between runs. Stuffing the whole KB into the system prompt (c) blows the context window and cost, and degrades quality as irrelevant text dilutes the signal. A one-time LoRA on a snapshot (d) is immediately stale for daily-changing data and still can't cite sources. Fine-tuning earns its place for behavior/format/tone, not volatile facts.

Provider structured-output features (e.g. JSON Schema-constrained decoding) restrict the token sampler at each step to tokens that keep the output valid against the schema, so the result is guaranteed-parseable and schema-conformant by construction. A free-text instruction (a) is best-effort: the model can still emit prose, trailing commas, or extra keys. Temperature 0 (c) only reduces variability — a deterministic output can be deterministically malformed. Retry-on-error (d) is a reasonable fallback and improves reliability, but it adds latency and cost and still has no hard guarantee on any single attempt; constrained decoding removes the failure mode at the source.

Function/tool calling is a protocol, not remote execution: given your tool definitions (name, description, JSON-Schema parameters), the model decides whether to call one and emits a structured tool-call with arguments. Your code executes the tool, then sends the result back as a tool/function message so the model can use it in its next turn. The provider does not run your code (a) — it has no access to your database or APIs. The model returns structured arguments, not executable code to eval (c); eval-ing model output would be a serious injection risk. And it is a concrete, parseable call, not a vague wish (d).

This is prompt injection: an LLM processes its entire context as one token stream and has no robust, built-in boundary between trusted instructions and untrusted data, so adversarial text embedded in fetched content can hijack behavior. The defense is architectural — keep untrusted content clearly delimited, apply least-privilege to tools, and require human approval or hard policy checks for dangerous actions like sending email or exfiltrating secrets — not a single magic setting. Temperature (a) controls randomness, not whether instructions are followed. Lengthening the system prompt (c) does not create a real trust boundary; a sufficiently crafted injection can still override it. Embedding dimensionality (d) is about retrieval quality and is unrelated to the model obeying injected commands.

Prompt caching stores the processed key/value state for a stable prefix (your unchanging system prompt + policy doc); on later requests the model reuses it instead of re-processing those tokens, which lowers input cost and time-to-first-token while leaving outputs identical — ideal when a large prefix is constant and only the tail varies. Changing temperature (b) affects randomness, not cost. Retrieving the system prompt via vector search (c) adds machinery and a retrieval step to fetch text you already have verbatim — pure overhead. Raising max_tokens (d) only sets an upper bound on output length; it does not reduce the cost of re-reading the same input every call and can increase cost if it lets responses grow.

Retrieval quality is about getting the right chunks into context. A reranker (a) reorders the initial candidate set with a more expensive, more accurate cross-encoder so the best passages float to the top. Chunking strategy (b) directly affects whether a chunk contains a complete, embeddable idea rather than a fragment. Hybrid retrieval (c) catches cases where exact terms/IDs matter that dense vectors miss, and vice versa. A better-matched embedding model (e) improves the similarity signal at the source. Raising max_tokens (d) only changes how long the answer may be — it does nothing about which documents were retrieved, so it cannot fix retrieving the wrong material.

Streaming (a) doesn't change total compute but dramatically improves perceived speed by showing the first tokens immediately. Model routing / cascading (b) sends the bulk of easy traffic to a cheaper model, cutting average cost and latency while preserving quality on the hard tail. Caching (c) avoids paying for work you've already done. Trimming context and bounding output length (e) reduces both input and output tokens, which is where the bill and the time go. Indiscriminately padding every prompt with more few-shot examples (d) does the opposite — it inflates input tokens (cost and latency) on every call, and beyond a point adds no accuracy, so it is a regression, not an optimization.

Hallucination drops when the model is anchored to real evidence: retrieval-grounding with an instruction to answer only from context (a) constrains it to supported claims. An explicit "I don't know" escape hatch (b) gives the model a correct option other than fabricating, which it otherwise tends to avoid. Citation + verification (c) turns grounding into something you can check, catching claims that aren't actually supported. Raising temperature (d) increases randomness and makes confident fabrication more likely, not less. A bare instruction to "never hallucinate" (e) is wishful — the model has no reliable internal signal of its own factuality, so an unenforceable command doesn't change behavior in a measurable way; grounding and verification do.

MCP is an open, client-server protocol that gives a uniform way to expose tools, resources, and prompts to LLM hosts (a), and its central payoff is write-once/reuse-everywhere interoperability across compatible clients (b). It does not remove the security burden: a server runs with whatever access you give it and the data it returns enters the model's context, so an untrusted or compromised server is a genuine injection/exfiltration risk (c) — least privilege and review still apply. MCP does not replace function calling (d); it is the transport/standard through which tool definitions and calls flow — the model still decides which tool to invoke. And it offers no guarantee against malicious server output misleading the model (e); content returned over MCP is untrusted data like any other.

The context window bounds the total tokens the model attends to: system prompt + retrieved context + conversation history + the tokens it generates. If you stuff the input close to the limit, you starve the output and risk truncated completions. Budget explicitly — reserve headroom for max_tokens of output, and remember that long outputs cost both latency and window space.