An API endpoint is fast in testing but slow under production load. How do you investigate?
Quick Answer
Reproduce with realistic concurrency, then use data — server metrics (CPU, memory, GC, thread-pool queue), request tracing/APM, and the database's own view (slow query log, wait stats). The usual causes: blocking calls on async paths (thread-pool starvation), N+1 or missing indexes that only hurt at volume, connection-pool exhaustion, lock contention, chatty external calls without caching, and cold caches.
First I'd reproduce it properly — a load test with production-like concurrency and data, because “fast in testing” usually just means one user and a small table. Then I look at the server: if CPU is low but the thread-pool queue is growing, that's almost always blocking calls on an async path — someone called .Result or a synchronous database method — and the pool starves. If it's not that, I trace a single slow request with APM to see whether the time is in my code, the database, or an external call. Most often it's the database: N+1 queries or a missing index that's invisible at 10 rows and painful at 10,000. I'd also check the connection pool isn't exhausted and HttpClient is being reused. Then fix one thing and re-run the load test.
A structured approach:
- Reproduce — load-test with production-like concurrency and data volume. “Fast on my machine” usually means single-user with a tiny dataset.
- Look at the host — is it CPU-bound, memory/GC-bound, or waiting? A growing thread-pool queue with low CPU screams blocking I/O on async code (
.Result,.Wait(), sync DB calls). - Trace one slow request — APM (Application Insights, OpenTelemetry) to see where the time goes: app code, database, or an external HTTP call.
- Database — capture the actual queries (EF logs them), check for N+1, missing indexes, and wait stats. A query that does 5 ms × 1 row becomes a problem at 500 rows/req × 200 req/s.
- Resource pools — connection pool (default max 100) exhausted → requests queue;
HttpClientmisuse (new per call) → socket exhaustion. - Contention — a
lock, a singleton with a mutex, a hot row being updated by every request. - Fix one thing, re-test.