Inspect and replay deliveries

Every attempt to deliver an event is recorded against its endpoint. When a receiver was down, slow or wrong, the log tells you what happened and a replay re-drives the delivery on a fresh attempt budget.

Delivery states#

Delivery rows are a summary: delivery_id, event_id, event_type, status, attempt_number, next_retry_at, last_response_status, created_at and updated_at. They never carry the body, the URL or a secret. Timestamps on delivery rows are epoch milliseconds, as the API returns them.

List an endpoint's deliveries#

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

const page = await client.webhooks.deliveries.list(process.env.WEBHOOK_ID!, { limit: 10 })
for (const d of page.deliveries) {
  console.log(d.delivery_id, d.event_type, d.status, 'attempt', d.attempt_number, 'last status', d.last_response_status)
}
console.log(page.deliveries.length, 'deliveries, next_cursor:', page.next_cursor)

Find what failed#

The summary route aggregates the failing rows server-side: by default the last 24 hours and the statuses dlq, failed, blocked and retrying. truncated is true when the examined-row cap was hit, so older failures may be missing; narrow since in that case.

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

const failing = await client.webhooks.deliveries.summary(process.env.WEBHOOK_ID!, {
  since: Date.now() - 24 * 60 * 60 * 1000,
  status: ['dlq', 'failed', 'blocked', 'retrying'],
})
for (const d of failing.deliveries) console.log(d.delivery_id, d.status, 'last status', d.last_response_status)
console.log(failing.deliveries.length, 'failing deliveries', failing.truncated ? '(truncated)' : '')

Replay one delivery#

A replay resets the delivery to pending on a fresh attempt budget and re-sends the same event, with the same event id and a fresh signature timestamp. A delivery that already succeeded is refused with replay_conflict (409) unless you force it; so is one that is mid-attempt, which you can retry shortly.

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

// force re-sends even a delivered event; the receiver sees the same event id again.
const result = await client.webhooks.deliveries.replay(process.env.WEBHOOK_ID!, process.env.DELIVERY_ID!, { force: true })
console.log('replay', result.delivery_id, result.status)

Replay everything in a window#

After an outage, replay every failing delivery between two times in one call. The batch is bounded and idempotent: it returns matched, enqueued and skipped counts and a next_cursor when there is more, and in-flight deliveries are skipped. The default statuses are dlq and failed; including delivered requires force.

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

let cursor: string | undefined
do {
  const result = await client.webhooks.deliveries.replayAll(process.env.WEBHOOK_ID!, {
    from: Date.now() - 24 * 60 * 60 * 1000,
    status: ['dlq', 'failed'],
    cursor,
  })
  console.log('matched', result.matched, 'enqueued', result.enqueued, 'skipped', result.skipped)
  cursor = result.next_cursor ?? undefined
} while (cursor)

The retry schedule and the dead-letter queue#

A delivery that fails retryably is retried on a fixed backoff schedule: seven legs summing to about 33 hours, so eight attempts in total before it is dead-lettered.

attempt 1 fails  wait 5s    attempt 2
attempt 2 fails  wait 30s   attempt 3
attempt 3 fails  wait 5m    attempt 4
attempt 4 fails  wait 30m   attempt 5
attempt 5 fails  wait 2h    attempt 6
attempt 6 fails  wait 6h    attempt 7
attempt 7 fails  wait 24h   attempt 8
attempt 8 fails  dead-letter (status dlq)

• Retried: 408, 429 and 503 (honouring Retry-After, clamped to 24 hours), any other 5xx, transport errors and timeouts.
• Terminal, never retried: a 2xx succeeds; any other 4xx is a client error to fix on your side (failed); an SSRF block, a redirect or a bad scheme is refused outright (blocked).

hookchat webhooks update "$WEBHOOK_ID" --enabled

Next#