Reliable webhook retries require durable storage, bounded backoff, duplicate-safe processing, and a dead-letter queue for unresolved failures. A timeout doesn’t prove the receiver rejected an event; it might have committed the change before the connection broke.
For a SaaS business, that uncertainty can mean duplicate emails or missing account access. Start with durable acceptance, then separate delivery, business processing, and recovery.
Key Takeaways
- Acknowledge webhooks after durable intake, then process them asynchronously.
- Retry temporary failures with exponential backoff and jitter, within explicit attempt and age limits.
- Protect business actions against duplicates and stale events.
- Assign dead-letter events an owner, monitor their age, and replay them through controlled recovery.
Build Webhook Retries Around Durable Acceptance
Persist before acknowledging
On the sending side, create an immutable event ID. Write the event to a transactional outbox alongside the business change. Otherwise, a crash between committing a subscription update and enqueueing its notification can lose the webhook.
On the receiving side, validate the signature using the original request bytes. Then persist the event and its processing job atomically before returning success.
If storage fails, don’t acknowledge the event. An in-memory task cannot survive a process restart.
Separate receipt from completion
A successful HTTP response confirms acceptance, not completion of every downstream action. Workers should handle provisioning, billing updates, and notifications after intake.
Provider contracts differ. GitHub requires a successful response within 10 seconds and doesn’t automatically redeliver failed deliveries. Stripe automatically retries failed live-mode deliveries for up to three days with exponential backoff.
Therefore, your internal queue needs independent retries. Once you acknowledge an event, the sender may consider its job finished.
Classify Failures Before Scheduling Another Attempt
HTTP responses need an explicit interpretation. The following is a starting policy for an API you control, not a universal provider rule.
| Delivery outcome | Recommended handling |
|---|---|
2xx | Mark delivery accepted. |
| Connection timeout or temporary network failure | Retry because acceptance is uncertain. |
408, 429, or temporary 5xx | Retry within the delivery budget. |
400 or 422 | Stop automatic retries for unchanged, invalid content. |
401 or 403 | Pause and surface credential or permission remediation. |
404 or 410 | Treat as an endpoint configuration or removal problem. |
3xx | Record failure rather than following redirects automatically. |
Endpoint contracts can override individual decisions. For example, a documented 409 might indicate temporary contention or a permanent conflict.
Hookdeck’s retry design guidance covers response handling alongside backoff and failure recovery.
Avoid treating every network-related error as temporary. An expired certificate or broken endpoint configuration needs intervention. Repeating unchanged requests adds load without repairing the cause.
Bound Backoff, Timeouts, and Retry Limits
Spread attempts with exponential backoff and jitter
Exponential backoff increases the waiting interval after repeated failures. Cap that interval so events don’t disappear into excessively long waits.
Add jitter, a randomized adjustment to each delay. Without it, thousands of events can retry simultaneously after a shared outage.
When an endpoint supplies Retry-After, honor it where compatible with your delivery deadline. If the requested wait exceeds that deadline, record the conflict rather than retrying immediately.
Store next_attempt_at durably. Don’t depend on sleeping application processes to remember future work.
Limit both time and resource consumption
Set connection and total request deadlines separately. A receiver that accepts a connection but never responds must not occupy a worker indefinitely.
Bound deliveries by both attempt count and event age. Count limits contain repeated fast failures; age limits prevent stale notifications from circulating indefinitely.
Choose budgets according to the event’s usefulness and your published delivery contract. A time-sensitive alert and an accounting notification have different recovery needs.
Also limit concurrency per endpoint and tenant. Otherwise, one unavailable customer endpoint can consume capacity needed by healthy customers. Backoff reduces load, but concurrency limits protect shared workers during the initial failure burst.
Make Duplicate Events and Reordering Safe
Deduplicate messages and business actions
Use a unique constraint scoped to the event source, tenant, and immutable event ID. An attempt ID should change between deliveries; the event ID should remain stable.
However, two different events can describe the same business operation. Protect that operation separately, such as enforcing one entitlement grant per paid invoice.
Commit local business changes and completion records together. For external actions, use a durable outbox and a stable downstream idempotency key where supported.
Svix’s delivery guarantees overview explains how retries and duplicate handling fit together.
Resolve ordering through current state
Stripe doesn’t guarantee event ordering. A payment failure can arrive after a later successful payment.
Before restricting access, retrieve the current invoice or subscription state. If that lookup times out, retry the internal job rather than applying an access change based on uncertain information.
For systems you control, object versions can identify stale updates. Serializing work per subscription or account can also help, but it reduces parallelism.
Delivery status and billing status are separate facts. A failed notification doesn’t establish that a customer still owes money.
For append-only activity or usage records, preserve each valid event. Don’t discard historical facts merely because a newer event arrived first.
Put a Practical Delivery Policy Into Operation
A simple sender can use five states: queued, delivering, waiting, delivered, and dead_letter.
The following policy keeps transport decisions separate from business processing:
- Claim a due delivery with a worker lease. Keep its event ID stable and create a new attempt ID.
- Send the signed request within its connection and total deadlines. Record the response or transport error.
- On
2xx, mark delivery complete. On a retryable failure, schedule a jittered backoff unless the attempt budget or event deadline has expired. - Move exhausted deliveries and permanent failures to the dead-letter queue with a reason. Pause credential-related failures until remediation.
Persist state changes before releasing the worker’s lease. Recover abandoned leases so crashes don’t strand events.
If using Amazon SQS, distinguish queue receives from actual HTTP attempts. A worker can receive a message and crash before sending anything. Its maxReceiveCount therefore isn’t automatically your webhook retry count.
Give Dead-Letter Events an Owner and Recovery Context
A dead-letter queue, or DLQ, stores work that needs investigation after automatic processing stops. Didit’s retry and DLQ overview describes this recovery role.
Keep sender delivery failures separate from receiver processing failures. An accepted webhook can still fail later during provisioning or email delivery.
Each record should preserve the tenant, event ID, destination, schema version, attempt history, last failure, and timestamps. Store sensitive payloads securely; avoid copying credentials or full customer data into logs.
Monitor new dead-letter arrivals, oldest unresolved age, failure categories, and affected tenants. Assign an owner and escalation deadline, including for jobs stuck outside the DLQ.
Retention must support recovery. AWS recommends longer retention for an SQS DLQ than its source queue. For standard queues, expiration remains tied to the original enqueue time, so moving a message doesn’t grant a fresh retention window.
An empty DLQ alone doesn’t prove health. Also monitor accepted-but-unprocessed events and missed expected deliveries.
Replay Failed Events Without Repeating Side Effects
Repair and select before replaying
Fix the underlying issue before releasing failed events. Then select records by tenant, endpoint, event type, failure reason, and time window.
Inspect current business state before repeating state-changing actions. A canceled subscription shouldn’t receive an old renewal reminder. A paid invoice shouldn’t trigger another payment-failure workflow.
Separate replay authorization from ordinary viewing permissions. Recovery tools can trigger real customer-facing changes.
Reuse identity while refreshing delivery security
Preserve the original event ID and business idempotency key. Create new attempt metadata, and sign outgoing webhook deliveries with a fresh timestamp.
Don’t disable signature validation to make old requests pass. Internal replay should use authenticated access to trusted, durably stored events.
Begin with a small batch, rate-limit it, and monitor processing failures before expanding. Automatic retries may overlap manual replay, so both paths need the same deduplication ledger.
Keep deduplication records for at least your supported automatic and manual recovery windows. Permanent business uniqueness constraints protect effects beyond those windows.
Test Failures at the Commit Boundary
A successful first delivery proves little about retry safety. The most revealing test disconnects the response after the receiver commits its work. Retrying must produce one business effect.
Also test concurrent duplicate deliveries, a worker crash after intake, reordered payment events, and an expired webhook secret. Simulate downstream email outages and provider API timeouts separately from transport failures.
For replay, test revoked tenant permissions, changed endpoints, and events whose business meaning is now stale. Assert that recovery preserves tenant boundaries and doesn’t revive canceled work.
Finally, demonstrate that alerts reach an owner when a job remains unresolved. A visible queue without an actionable alert still leaves customers waiting.
Conclusion: Make Recovery Part of Delivery
Reliable webhook retries depend on durable intake, bounded attempts, and duplicate-safe business actions. A dead-letter queue completes that design only when someone monitors it and can replay events safely.
Start with one event type and test a response lost after commit. That boundary reveals whether your system can recover without charging twice, sending duplicate reminders, or granting access incorrectly.
