PrismitaLEARN · PRACTICE · GROW

A deployment has broken production. Walk me through what you do.

ASP.NET Core Scenario Advanced 3–5 Yrs5+ Yrs Scenario 2 min read

Quick Answer

Stabilise first, diagnose second. Confirm the blast radius, roll back to the last known-good version (or shift traffic away with blue/green), communicate status, then investigate calmly with the logs and the diff. Only forward-fix instead of rolling back if a rollback is impossible (e.g. an already-applied breaking migration). Afterwards: blameless post-mortem and a guardrail so it can't recur.

The instinct to debug it live is the wrong one — stabilise first. I confirm what's actually broken and how widespread it is, then roll back to the last good version, or if we're on blue/green I just shift traffic back to the old environment. That takes minutes and buys us time. I communicate what's happening and keep a timeline. The one thing that complicates a rollback is the database — if the deploy ran a breaking migration, rolling back the code isn't enough, which is the argument for backward-compatible schema changes. Once it's stable I investigate properly with the diff and the logs, fix forward with a small tested change, and then run a blameless post-mortem focused on what guardrail — a smoke test, a canary — would have caught it.

  1. Assess — what's broken, for whom, how bad? Check health checks, error rates, key user journeys.
  2. Stabilise — roll back to the previous release, or in blue/green flip traffic to the old slot. This is faster and safer than debugging live.
  3. Communicate — tell stakeholders and, if user-facing, post a status update. Keep a timeline.
  4. Special case: database — if the deploy applied a non-backward-compatible migration, a code rollback alone won't fix it. This is why schema changes should be expand/contract and backward-compatible.
  5. Diagnose — with pressure off, use the deployment diff, structured logs and APM to find the cause.
  6. Fix forward — a small, tested change; redeploy through the normal pipeline.
  7. Post-mortem — blameless, focused on how the process let it through (missing test, no canary, no smoke check) and what guardrail to add.
All ASP.NET Core questions More on Scenario