You get an intermittent NullReferenceException in production that you can't reproduce locally. How do you approach it?
Quick Answer
Get the exact stack trace and a correlation ID from structured logging, and identify the line and the reference that's null. “Intermittent” and “only in production” usually points to concurrency (a shared/captured DbContext or a non-thread-safe singleton), data that only exists in prod (a null column, a missing related row), caching/config differences, or a race between requests. Add targeted logging, then reproduce with a test that mimics the concurrency or the data.
The first thing I need is the actual stack trace with a correlation ID, from structured logging or an APM tool — a plain “object reference not set” in a log file is useless. That tells me the method and line, and I work out which reference there can be null. Then I think about why it's intermittent and prod-only. The big one is concurrency — a DbContext that's been captured in a singleton or used from parallel tasks will throw random nulls under load. After that it's data that only exists in production, like a nullable column that's always filled in my test data, or a related row that got deleted. Or a config value that's in my dev appsettings but not set in the prod environment. I add logging around the suspect line, ship it, and once I know the trigger I write a test that reproduces it and fix it against that.
- Capture the exception properly — full stack trace, the request path, user, inputs, and a correlation ID tying together all log lines for that request. Application Insights / Sentry / Seq make this trivial; a bare log message doesn't.
- Pin the line — the stack trace gives you the method and line; work out which of the references there can be null.
- Form hypotheses by category:
- Concurrency — a
DbContextshared across threads (captured in a singleton, stored in a static, or used from parallel tasks) throws odd nulls and “second operation” errors intermittently. - Production-only data — a nullable column that's always populated in your seed data; an orphaned row; a deleted parent.
- Timing / ordering — a cache populated by a background task that a request beats; lazy init without a lock.
- Config — a setting present in dev
appsettingsbut not in the prod environment, so a service resolves to null.
- Concurrency — a
- Add defensive logging around the suspect line (log the inputs and which reference is null) and ship it.
- Write the failing test — once you know the trigger, reproduce it with a concurrency test or a row with the prod-shaped data, then fix and keep the test.