DevOps Interview Questions

Canary releasing exists precisely for this scenario: you expose the new version to a small slice of real traffic (say 1–5%), compare its error rate, latency, and business metrics against the stable version, and only ramp up if it holds. A regression is therefore detected while it harms a fraction of users, and you can roll back by shifting that slice back. Blue-green (b) is excellent for instant cutover and instant rollback, but at the moment of cutover the new version takes the entire load, so a load-sensitive regression hits everyone — it minimizes downtime, not blast radius under live traffic. Rolling updates (c) do gradually replace instances, but the claim is wrong: as soon as a new instance is in the pool it serves real traffic, and rolling gives you no metric-gated traffic split. (d) is false — the whole point of these strategies is progressive exposure without a downtime window.

Both practices keep the codebase in a continuously releasable state — every change is automatically built, tested, and proven deployable. The single distinction is the production gate. Continuous delivery stops just short of production: the artifact is release-ready and could ship at the click of a button, but a human decides when. Continuous deployment removes that button — any change that passes the full automated pipeline is released to production with no manual step. Option d inverts the definitions. Option b confuses these with continuous integration (frequent merges plus automated testing on each commit), which is the upstream practice both build on. Option c is wrong: continuous delivery is about being able to release to prod at any time, not deploying only to staging, and the two terms are genuinely different, not a rename. The interview tell is naming the manual-approval gate as the one differentiator.

The three pillars are logs, metrics, and traces, and each answers a different question. Metrics are cheap aggregate time-series — great for 'is latency up?' and dashboards/alerts, but they aggregate away the per-request path, so they tell you that something is slow, not where in a multi-service call. Logs are discrete events rich in detail, but on their own they are per-service and don't automatically reconstruct one request's journey across services. Distributed tracing is purpose-built for this: a trace ties together the spans of a single request as it crosses service boundaries, each span timed, so you can read off exactly which downstream hop dominates the latency. Option b overstates what a single metric can do; option c assumes logs auto-correlate across services (they don't without a shared trace/correlation id); option d is wrong — alerting is built on metrics/logs, it is not a fourth pillar and does not localize a slow span.

Immutable infrastructure means you never modify a provisioned server after it boots. To change anything — a patch, a new dependency, a config tweak — you bake a new image (AMI/container/VM template), launch fresh instances from it, shift traffic, and terminate the old ones. The change is captured once, in the image build, so every instance from that image is byte-for-byte identical and fully reproducible. This directly attacks configuration drift and 'snowflake servers': the gradual, undocumented divergence that happens when humans and agents make ad-hoc in-place edits over months until no two hosts match and none can be rebuilt from source. Option b (in-place SSH patching) is exactly the mutable anti-pattern that produces drift — manual changes are rarely uniform or recorded. Option c describes convergent configuration management (Puppet/Chef-style), which mutates live hosts and is the opposite of immutable. Option d misreads 'immutable' as a read-only filesystem; it's about not re-provisioning live hosts, not about disk writes.

An error budget is derived directly from the SLO: it is 100% minus the objective, so a 99.9% target permits 0.1% unreliability over the measurement window (a). That budget is a currency. While it is unspent the team has room to move fast — deploy often, take calculated risk — and when it runs out, the agreed error-budget policy kicks in: stop shipping risky changes and redirect effort to reliability until the service earns budget back (b). Its real organizational value is making the eternal velocity-vs-stability tension an objective, shared number instead of a turf war between developers who want to ship and ops who want stability (c). Option d is the classic trap: a budget that is never spent means your SLO is too loose (or you're over-investing in reliability users don't perceive) — you're leaving velocity on the table, so consistently spending zero is a signal to recalibrate, not a victory. Option e is wrong: the budget is computed from the SLO/SLI math, not by tallying incident counts.

True, and this decoupling is the main strategic reason teams adopt feature flags. Deploying ships code to the servers; releasing exposes behaviour to users. A flag is a runtime switch wrapping the new code path, so you can merge to trunk and deploy continuously — keeping branches short-lived — while the feature stays off ('dark') in production. You then enable it on your own schedule: for internal users first, for a 5% canary cohort, or for everyone, and you can kill it instantly without a rollback deploy if it misbehaves. This is what makes trunk-based development and continuous deployment practical for large, in-progress features, and it underpins experimentation (A/B tests) and progressive delivery. The cost is real and worth naming: flags are conditional logic that accumulate, so undisciplined teams drown in stale flag debt — short-lived flags should be removed once a feature is fully rolled out.

Object spread merges left to right, and on a key collision the last spread wins. { ...defaults, ...process.env } therefore lets any variable present in the real environment override the committed default — so DATABASE_URL resolves to postgres://prod-db/app. This is the twelve-factor 'store config in the environment' principle in miniature: the same artifact ships everywhere, and per-environment values (prod DB, secrets, feature toggles) come from the environment at runtime, while a committed defaults file gives a sane local-dev fallback that never contains real secrets. Option b inverts the precedence (committed defaults are the fallback, not the override). Option c is wrong — duplicate keys across the two sources is the normal, intended case, not an error. Option d is false: process.env is a readable map here and nothing about reading it from a file changes merge order. The interview point is that environment beats baked-in defaults, which is what keeps one build deployable to every environment.

The whole value of building once and promoting an immutable, SHA-tagged artifact shows up at rollback time. The script looks up the artifact that was running two deploys ago (app:git-2b7e08) and re-deploys those exact bytes — no recompilation, no dependency resolution, no fresh image build. That makes the rollback fast and, crucially, reproducible: you are restoring the artifact that was already proven in prod, not a hopefully-equivalent rebuild. Option b describes the rebuild-per-environment anti-pattern, which the prompt explicitly rules out — rebuilding from an old commit can silently pull newer transitive dependencies or a newer base image and yield a different binary, defeating the point of rolling back. Option c is the opposite of true: immutability is what enables clean rollback by tag. Option d is the important caveat stated as a wrong answer — re-deploying an artifact rolls back code only; database migrations are not reversed by this and must be handled separately (which is why backward-compatible, expand-then-contract migrations matter).

Line B is structured logging: a machine-readable object with named, typed fields. A log platform parses it into indexed fields (user_id as a number, amount as a number), so a query like event:checkout AND amount > 100 is exact and survives wording changes. Line A is a human sentence — to extract the amount you'd need a fragile regex that breaks the moment someone changes 'checked out for $' to 'purchased', drops the dollar sign, or localizes the message, and the value arrives as text, not a comparable number. That brittleness is exactly why production systems standardize on structured logs with consistent field names (and a correlation/trace id) across services. Option b has it backwards; prose is easy for humans, hard for machines. Option c is wrong because structure is precisely what makes reliable field queries possible. Option d is false — you absolutely can threshold over a parsed numeric log field; metrics are often derived from such logs, but the log field itself is queryable.

Idempotency means running the operation any number of times leaves the system in the same end state as running it once — the contract a convergent config tool depends on, since it re-applies on every run. This script breaks that contract twice. useradd app is a create that fails with a non-zero exit when the user already exists; under set -e that aborts the whole converge on the second run. And echo ... >> /etc/app.env appends, so each run adds another identical CONFIG_DIR=/etc/app line, growing the file and potentially changing how the app parses it. The fix is to make every step check-then-act so it's safe to repeat: guard the user creation (id app >/dev/null 2>&1 || useradd app) and add the config line only if it isn't already present (grep -qxF 'CONFIG_DIR=/etc/app' /etc/app.env || echo 'CONFIG_DIR=/etc/app' >> /etc/app.env). Option b is backwards — set -e correctly surfaces the latent failure; deleting it just hides a broken converge. Option c fixes only half the problem and wrongly calls the append idempotent. Option d is plainly false on both counts.

Blamelessness shifts the postmortem's question from 'who caused this?' to 'what about our systems and processes made this failure possible and its blast radius large?'. It rests on the assumption that people generally do their best with the context they had, so a human error is treated as a symptom of a system that allowed it (a missing guardrail, a confusing UI, an alert that fired too late), not as the cause to be punished. The reliability payoff is mechanistic, not soft: blame creates fear, fear suppresses honest reporting, and suppressed reports hide the real contributing causes — so the latent fault recurs. Remove blame and you get psychological safety, which produces candid, detailed accounts that reveal the true systemic gaps, which in turn become concrete, tracked action items (automation, safer defaults, better detection) that actually prevent recurrence. A strong answer stresses systems-over-individuals, psychological safety enabling honest reporting, and that accountability is redirected to fixing the system, not removed.

Interviewers probe secret hygiene because the failure modes are common and costly. The headline anti-pattern is committing secrets to version control: git history is permanent, so a secret that ever touched a commit is exposed via every clone, fork, and the reflog even after you 'delete' it — which is why a leaked secret must be revoked and rotated, not merely removed from HEAD. Adjacent anti-patterns are hardcoding secrets into the application image or build args (every artifact copy now carries them), logging them, and reusing one un-rotated static credential everywhere. The sound model: keep secrets out of code and out of the build artifact, store them in a dedicated vault or cloud secret manager, inject them at deploy/runtime through environment variables or mounted files, scope them per environment with least-privilege access and an audit trail, and make them rotatable — ideally short-lived or dynamically issued so a leak has a small blast radius. A strong answer pairs the 'never in git/never in the image' rule with vault + runtime injection + rotation/revocation.

The interview point is the pull-based reconciliation loop versus push-based deployment. In GitOps, declarative manifests in git define desired state and an in-cluster controller continuously diffs live state against the repo and converges to it — making git the audit log, the rollback mechanism (revert the commit), and the drift-correction engine all at once. Traditional CI/CD pushes: the pipeline ends by pushing artifacts/changes into the environment using credentials the external system holds. Strong answers name (1) git as single source of truth, (2) a controller that reconciles continuously, (3) the security win of pull-from-inside vs handing prod creds to CI, and (4) automatic drift detection. A common follow-up is the limitation: GitOps assumes a declarative target and a healthy reconciler, so it maps cleanly onto Kubernetes but awkwardly onto imperative provisioning.

Incident response follows a deliberate order. First you must detect the problem — ideally an automated alert wired to an SLI breach, not a customer tweet. Then triage and declare: size up impact and severity, formally declare an incident so the right people engage, and name an incident commander to coordinate (separating coordination from hands-on debugging). Next comes the phase juniors most often get wrong: mitigate before you diagnose. The first duty is to stop user pain by the fastest safe lever — roll back the last deploy, fail over to a healthy region, or disable the feature flag — even if you don't yet understand the root cause. Only once the bleeding stops do you resolve: verify the SLI is back in bounds and stand the incident down. Finally, after the adrenaline fades, the postmortem extracts lasting value through a blameless review that identifies contributing causes and produces tracked action items so the same failure can't recur. The two classic mistakes are chasing root cause before mitigating, and skipping the postmortem so nothing is learned.

This is the foundational IaC distinction. A declarative approach states what you want — 'three web servers behind a load balancer with this config' — and the tool diffs that desired state against reality and figures out how to converge (create what's missing, change what differs, leave correct things alone). Because the spec is the end state, re-applying it is naturally idempotent and the tool can detect and correct drift. An imperative (or procedural) approach states how: an ordered list of commands ('create server, then install package, then edit file') that you must make safe to re-run yourself, since the script doesn't inherently know the current state. Most modern IaC and orchestration tools are declarative for exactly these reasons — idempotency, drift detection, and reproducibility — while shell-style provisioning scripts are imperative. Knowing which model a tool uses tells you how it will behave on a second run.