Webhooks you can trust: idempotency, ordering, and the retries you didn't plan for

Several of the marketplace integrations we've worked on are webhook-driven - the platform calls us when an order is placed, a listing is acknowledged, a product changes. Webhooks are simpler than polling right up until traffic picks up, and then every assumption a naive receiver makes turns into a bug.

Verify it's really them, before anything else

A public webhook URL will be found and probed. Every serious provider signs its payloads - typically an HMAC of the raw body with a shared secret, in a header. Recompute it over the exact bytes you received, not the re-serialised JSON, and reject on mismatch. If the provider signs a timestamp too, reject anything older than a few minutes so a captured request can't be replayed later.

Acknowledge fast, process afterwards

Providers expect a 2xx within a few seconds. Do the signature check, write the raw event to a queue or a table, and return 200. Do the real work - updating orders, calling other systems - in a separate worker. If you process inline and it runs long, the provider times out, marks the delivery failed, and retries, and now you're doing the same work twice under load.

Assume every event will be delivered more than once

Retries, provider-side bugs, and your own timeouts all produce duplicates. Every event has an id; store processed ids and check before acting. Better, make the effect idempotent - "set order X to shipped" applied twice is harmless; "add a shipping fee to order X" applied twice is a support ticket. Where you can't make it naturally idempotent, the dedupe table is the safety net.

Don't trust the order they arrive in

• "Order updated" can arrive before "order created" - parallel delivery and retries reorder events routinely.
• Each event usually carries a timestamp or a version; apply it only if it's newer than the state you already hold for that entity.
• If the matching create hasn't arrived yet, park the update briefly and retry rather than dropping it or writing a half-record.

Give up deliberately, not silently

A worker that keeps retrying a permanently broken event blocks the queue behind it. Cap the attempts, then move the event to a dead-letter store with the error and the full payload, and alert. A dead-letter queue you actually watch is the difference between "we replayed six events" and "we've been dropping cancellations for a week".

Keep a poll as a backstop

Even a well-built receiver misses events - an outage on either side, a deploy at the wrong moment. Once a day, pull the last day or two of records from the provider's API and diff against what you processed. It isn't the primary path; it's the net that catches what the primary path drops.

Our JWT Debugger decodes the signed tokens some providers use for webhook auth; the Hash Generator lets you reproduce an HMAC signature by hand when you're debugging a rejection; the JSON Formatter makes a raw payload readable; and the Timestamp Converter turns the epoch values in an event into something you can line up against your logs.

The webhook that breaks you in production is never the one you tested. It's the redelivery of an event from four minutes ago, arriving while you're mid-deploy.

How do you handle duplicate webhook deliveries?

Treat duplicates as expected, not exceptional. Give every event an id, record the ids you've processed, and skip any you've seen before. Where possible, make the action itself idempotent so applying it twice changes nothing. For actions that aren't - adding a charge, incrementing a count - the processed-id check is what stops the second delivery doing damage.