The Case Against Webhook Retry Libraries

Your app sends webhooks. Sometimes they fail. The obvious fix: add a retry library.

import retry from 'async-retry'

await retry(async () => {
  const res = await fetch(webhookUrl, { method: 'POST', body: payload })
  if (!res.ok) throw new Error(`Failed: ${res.status}`)
}, { retries: 5, minTimeout: 1000 })

Problem solved, right?

You’ve just created three new problems.

Problem 1: You’re Blocking

That retry loop runs in your request handler. While waiting for retries, you’re holding a connection open. Your server is doing nothing useful. If the webhook endpoint is slow or down, your own app’s latency suffers.

The standard fix: move webhook delivery to a background job. Now you need a job queue. Redis. Workers. Monitoring. The “simple” retry library just became infrastructure.

Problem 2: You’re Stateless

Your server restarts. Mid-retry, everything in memory is gone. The webhook that was on retry attempt 3 of 5? Lost. You’ll never know it failed permanently. Your customer’s integration silently breaks.

The standard fix: persist retry state to a database. Track attempts, next retry time, failure count. Add a cron job to process the retry queue. More infrastructure. More failure modes.

Problem 3: You’re Not Backpressure-Aware

Retry libraries use exponential backoff. Good. But they don’t coordinate across your fleet. Ten servers all retrying the same failed endpoint hammer it with retry storms. You’re making the problem worse.

The standard fix: distributed rate limiting. Shared state for retry coordination. More infrastructure.

The Pattern That Emerges

Every “just add a retry library” turns into:

1. Retry library
2. Background job queue
3. Persistent storage for retry state
4. Monitoring for stuck jobs
5. Coordination across servers
6. Cleanup for abandoned retries

You’ve built a webhook delivery system. You didn’t mean to. You just wanted reliable delivery.

What Reliable Delivery Actually Requires

Webhook delivery at scale needs:

• Durability: Retry state survives server restarts
• Isolation: Each webhook retries independently
• Backpressure: Failed endpoints don’t cascade to others
• Observability: Know what’s failing and why
• Cleanup: Completed webhooks don’t accumulate cruft

This is infrastructure. Not a library. Not a few dozen lines of retry logic.

The Relay Approach

Each webhook gets its own Durable Object. Timer and state live together. Nothing is in-memory; everything is persistent by default.

curl -X POST https://relay.solenoid.systems/v1/relay/schedule \
  -H "Authorization: Bearer sm_your_api_key" \
  -d '{
    "url": "https://partner-api.com/webhook",
    "body": {"event": "order.completed", "order_id": "12345"},
    "delay": "0s"
  }'

delay: "0s" means “send now.” The webhook goes out immediately. If it fails, Relay retries with exponential backoff. If it keeps failing, you get notified. Either way, your server returned 200 to your user instantly.

Failure Handling

5xx errors: Transient failure. Retry with backoff. Cloudflare’s Durable Object alarm system handles the scheduling. No cron jobs.

4xx errors: The request is bad. Retrying won’t help. Relay stops immediately and records the failure. No wasted retries.

Timeouts: The endpoint didn’t respond in time. Treated as transient. Retry with backoff.

Permanent failure: After maximum retries, the webhook is marked failed. You decide what happens next—alert, fallback, ignore.

What You Don’t Run

With Relay:

• No Redis cluster
• No job workers
• No retry state table
• No cron jobs for retry processing
• No monitoring for queue depth
• No cleanup jobs for old entries

You make an API call. The webhook delivers or you hear about it. That’s the contract.

When Libraries Make Sense

Retry libraries work fine when:

• Failures are rare and brief
• You can tolerate lost retries on server restart
• Your webhook volume is low (hundreds/day, not thousands/hour)
• You’re already running job infrastructure for other reasons

If your webhooks are “fire and mostly forget,” a library is fine. If you’re promising delivery to customers, if integrations depend on reliable webhooks, if failure costs are high—that’s when you need infrastructure.

The Cost Comparison

Library approach total cost:

• Redis: ~$50/month for managed, more for HA
• Worker servers: ~$20/month minimum
• Monitoring: Built into your existing stack (hopefully)
• Engineering time: Setup, debugging, maintenance

Relay cost:

• Free: 10,000 webhooks/month to validate
• Starter ($19.99/mo): 100,000 webhooks
• Pro ($49.99/mo): 500,000 webhooks
• Scale ($149.99/mo): 2,000,000 webhooks

Zero infrastructure. Zero monitoring. Zero maintenance. If you’re sending more than a few thousand webhooks per month, the infrastructure savings alone justify the cost.

Try It

Replace one webhook with Relay. See if delivery improves. See if your ops burden decreases.

# Your current code
await fetch(webhookUrl, { body: payload })

# With Relay
await fetch('https://relay.solenoid.systems/v1/relay/schedule', {
  headers: { 'Authorization': 'Bearer sm_your_key' },
  body: JSON.stringify({ url: webhookUrl, body: payload, delay: '0s' })
})

Same result for the happy path. Better result for everything else.