INCIDENT
Incident: How a PostgreSQL connection pool spike crashed our checkout worker for 40 minutes
Incident Summary
During our mock placement rush last Sunday, our worker service began failing with FATAL: remaining connection slots are reserved for non-replication superuser connections.
Timeline & Root Cause
1. 08:14 UTC: Worker traffic scaled from 1,200 to 9,400 concurrent requests.
2. 08:18 UTC: An unindexed foreign key join on candidate_profiles caused queries to lock rows for ~420ms instead of ~4ms.
3. 08:22 UTC: Database connections climbed from 28 to the PostgreSQL ceiling of 100.
4. 08:35 UTC: Cascading HTTP 504 timeouts across the Truth Teller sync queue.
Resolution & Long-Term Fix
transaction pooling mode, capping backend connections to 25 while multiplexing 800+ client connections.(user_id, status) and enforced strict 2-second query cancellation timers:sql
ALTER ROLE jobmint SET statement_timeout = '2500ms';
Has anyone run into connection starvations with serverless Prisma or Drizzle under Next.js 15?
0 Discussions
Join Technical Discussion
Threaded Responses (0)
No replies yet. Be the first to share an engineering perspective.