Webhooks Operations & Monitoring

Monitoring webhook health is critical for maintaining reliable integrations. Catena provides comprehensive metrics, detailed logs, and replay capabilities to help you track delivery performance and recover from failures.

Circuit Breaker

Catena automatically pauses webhooks when delivery performance drops below acceptable thresholds. This prevents wasted resources and signals that immediate attention is needed.

When a webhook is paused:

• Event delivery stops
• New events are queued but not delivered
• Webhook status changes to stale

You can subscribe to webhook.staled events to be notified automatically when the circuit breaker trips.

You can also query for paused webhooks directly:

curl 'https://api.catenatelematics.com/v2/notifications/webhooks?status=stale' \
  -H 'Authorization: Bearer <token>'

The circuit breaker uses different evaluation strategies depending on delivery volume:

Reactivation

After identifying and fixing the root cause of delivery failures, reactivate the webhook to resume event delivery:

    curl -X POST \
      --url https://api.catenatelematics.com/v2/notifications/webhooks/<webhook_id>/activate \
      -H 'Authorization: Bearer <token>'

Delivery Metrics

The Notifications API provides real-time and historical metrics to monitor webhook performance and reliability across four rolling time windows: 6h, 24h, 7d, and 14d.

curl --url https://api.catenatelematics.com/v2/notifications/webhooks/<webhook_id>/metrics \
  -H 'Authorization: Bearer <token>'

{
  "webhook_id": "247b2dea-a030-48b7-9a05-ee33c1b6ab0a",
  "http_attempts": {
    "6h": 105,
    "24h": 420,
    "7d": 3000,
    "14d": 6500
  },
  "http_success_attempts": {
    "6h": 100,
    "24h": 400,
    "7d": 2800,
    "14d": 6000
  },
  "http_failure_attempts": {
    "6h": 5,
    "24h": 20,
    "7d": 200,
    "14d": 500
  },
  "message_count": {
    "6h": 100,
    "24h": 400,
    "7d": 2800,
    "14d": 6000
  },
  "message_success_count": {
    "6h": 98,
    "24h": 390,
    "7d": 2700,
    "14d": 5800
  },
  "success_rate": {
    "6h": 98,
    "24h": 97,
    "7d": 96,
    "14d": 96
  },
  "avg_response_time_ms": {
    "6h": 150,
    "24h": 160,
    "7d": 155,
    "14d": 158
  },
  "ewma_success_rate": 0.97,
  "dlq_count": 3
}

Each time-windowed field returns values for four rolling windows: 6h, 24h, 7d, and 14d. Key fields to monitor:

• success_rate — The percentage of messages successfully delivered after all retries. If any window drops below ~95%, investigate immediately.
• ewma_success_rate — The exponentially weighted moving average success rate. This is what the circuit breaker evaluates for high-volume webhooks — if it falls below 0.95, the webhook will be marked stale.
• http_attempts vs message_count — HTTP attempts includes retries; message count counts unique events. A high ratio of attempts to messages indicates frequent retries.
• avg_response_time_ms — Sustained values above 2500ms put you at risk of timeouts.
• dlq_count — Any non-zero value means events need to be replayed before the 14-day retention window expires.

Delivery Logs

Access detailed logs for every webhook delivery attempt. Logs are available for up to 14 days and include status codes, response times, error messages, and per-attempt detail.

curl --url "https://api.catenatelematics.com/v2/notifications/webhooks/<webhook_id>/logs?status=failed" \
  -H 'Authorization: Bearer <token>'

{
  "logs": [
    {
      "created_at": "2026-01-15T10:30:00.596772Z",
      "webhook_id": "247b2dea-a030-48b7-9a05-ee33c1b6ab0a",
      "message_id": "62cb8fea-e017-4b08-86b7-4469fa872b91",
      "event_name": "vehicle_location.added",
      "status": "failed",
      "status_code": 504,
      "error_message": "Request timeout. Please ensure the webhook endpoint is reachable and acknowledges receipt in less than 3 seconds.",
      "response_time_ms": 3000
    },
    {
      "created_at": "2026-01-15T09:14:22.312445Z",
      "webhook_id": "247b2dea-a030-48b7-9a05-ee33c1b6ab0a",
      "message_id": "91fa2bcd-3301-4e7a-bc12-7734ab991e22",
      "event_name": "vehicle.modified",
      "status": "failed",
      "status_code": 502,
      "error_message": "Connection error: Unable to establish connection to webhook endpoint.",
      "response_time_ms": null
    }
  ],
  "total": 2,
  "page": 1
}

Event Replay

Recover from delivery failures by replaying events from the Dead Letter Queue (DLQ). When events fail all automatic retry attempts, they’re stored in the DLQ for up to 14 days, giving you time to fix issues and replay them.

The replay functionality redelivers all DLQ events for a webhook subscription, allowing you to recover from temporary outages, application bugs, or configuration issues without losing data.

Common Replay Scenarios

How to Replay Events

    curl -X POST \
      --url https://api.catenatelematics.com/v2/notifications/webhooks/<webhook_id>/replay \
      -H 'Authorization: Bearer <token>'

Monitoring Best Practices

Troubleshooting