9:15 AM Daily Production Incident•Beyond LeetCode: Real Engineering Outages
Next Incident Drop in:05h 12mBrowse 9 AM Drops →
RZPRazorpay Payments

Double-Charge Outage: Payment Webhook Race Condition

3:00
3-Minute SLA

PagerDuty Alert at 09:14 AM: Users are receiving duplicate order confirmations and wallet debits! Upstream payment gateways retry webhooks within 200ms when high network jitter occurs. Concurrent webhook delivery causes the order status check to pass twice before the database lock acquires.

Production Impact: ₹14.2 Lakhs in duplicate order authorizations across 850 merchants
datadog-apm-live-tail // razorpay-webhook-race-condition● Streaming Logs
[09:14:02.120] POST /webhooks/razorpay/payment.captured - Event: pay_89f02a [Worker-1]
[09:14:02.145] POST /webhooks/razorpay/payment.captured - Event: pay_89f02a [Worker-4] (Gateway Retry)
[09:14:02.160] [Worker-1] Checking order status: order_9941 -> PENDING
[09:14:02.162] [Worker-4] Checking order status: order_9941 -> PENDING (Race Condition!)
[09:14:02.210] [Worker-1] Fulfilling order & capturing credits: +₹2,499
[09:14:02.215] [Worker-4] Fulfilling order & capturing credits: +₹2,499 (DUPLICATE CAPTURE!)
[09:14:02.220] 🚨 FATAL: Duplicate fulfillment detected for order_9941. Ledger mismatch!
Principal Engineer Hint:

Use atomic idempotency check via Redis distributed lock or SQL row-level lock before processing.

buggy_service.tsRoot Cause Identified
// ❌ BUGGY WEBHOOK HANDLER
export async function handlePaymentWebhook(req, res) {
  const { event, orderId, paymentId } = req.body;

  // Query order state
  const order = await db.orders.findById(orderId);
  
  if (order.status === "FULFILLED") {
    return res.status(200).send("Already fulfilled");
  }

  // ⚠️ Gap: Async delay allows concurrent retry to pass the check!
  await sleep(40); 
  
  await db.orders.update(orderId, { status: "FULFILLED", paymentId });
  await creditMerchantWallet(order.merchantId, order.amount);

  return res.status(200).json({ success: true });
}
SELECT PRODUCTION HOTFIX PATCH:+75 XP & Proof-of-Work
Atomic Redis SETNX Idempotency Lock
// ✅ HOTFIX: Atomic Distributed Lock
export async function handlePaymentWebhook(req, res) {
  const { event, orderId, paymentId } = req.body;
  const lockKey = `lock:webhook:${paymentId}`;

  // Atomic lock: Only 1 worker acquires lock for 30 seconds
  const acquired = await redis.set(lockKey, "1", "NX", "EX", 30);
  if (!acquired) {
    return res.status(200).send("Event already being processed by another worker");
  }

  try {
    const order = await db.orders.findById(orderId);
    if (order.status === "FULFILLED") return res.status(200).send("Fulfilled");

    await db.orders.update(orderId, { status: "FULFILLED", paymentId });
    await creditMerchantWallet(order.merchantId, order.amount);
    return res.status(200).json({ success: true });
  } finally {
    await redis.del(lockKey);
  }
}
Increase sleep delay to 100ms
Drop all retried webhooks silently by timestamp