--- title: Handle jobs and events description: Register a webhook, verify its events, deduplicate deliveries, and recover missed events. doc_version: 0.1.0-preview last_updated: 2026-09-14 --- # Handle jobs and events Register a webhook to get signed status events. Verify each POST with a Standard Webhooks library, deduplicate events by ID, and recover missed events from the delivery log or with a status read. ## Set up a receiver Create an HTTPS endpoint in your application. Then call [Create webhook](/engine/reference/create-webhook) with a name, a scope, and the endpoint URL. The answer holds the signing `secret`. Engine shows it once, so store it in your server secret store before you continue. **POST https://engine.example.com/webhooks** ```http POST /webhooks HTTP/1.1 Host: engine.example.com Authorization: Bearer Content-Type: application/json { "name": "acme-status", "scope": { "kind": "namespace", "namespaces": [ "acme" ] }, "url": "https://app.example.com/engine-events", "types": [ "job.updated", "file.status_changed", "file.stage_changed", "collection.status_changed" ] } ``` **201 Created · illustrative secret placeholder** ```json { "id": "whk_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "name": "acme-status", "scope": { "kind": "namespace", "namespaces": [ "acme" ] }, "url": "https://app.example.com/engine-events", "types": [ "job.updated", "file.status_changed", "file.stage_changed", "collection.status_changed" ], "enabled": true, "disabled_reason": null, "signing": { "secret_fingerprint": "sha256:9b2d44f0", "previous_secret_fingerprint": null, "previous_secret_expires_at": null }, "delivery": { "pending_deliveries": 0, "last_succeeded_at": null, "last_failed_at": null }, "secret": "whsec_", "created_at": "2026-10-06T12:00:00Z", "updated_at": "2026-10-06T12:00:00Z" } ``` The secret starts with `whsec_`. It signs the POSTs to this webhook and authenticates nothing on the Engine API, so your receiver holds no Engine key. `types` selects the event types. Omit it, or send `null`, for every type. Each webhook has its own delivery order, so a second webhook for `research.updated` and `research.progress` never waits behind file events. Engine sends no POST at create, so the receiver need not be up yet. Call [Test webhook](/engine/reference/test-webhook) to send one signed `webhook.test` POST and read the receiver’s answer. The webhook gets only the events that occur after its create. [Rotate webhook secret](/engine/reference/rotate-webhook-secret) replaces the secret. The previous secret keeps signing for an overlap, 24 hours by default, so you can deploy the new secret at any time in it. Use an overlap of 0 for a leaked secret. ## Choose who receives events A webhook gets only the events in its scope and its types. A webhook’s scope works like a key’s scope. | Scope | Body | Gets the events of | | --- | --- | --- | | `cluster` | `{"kind": "cluster"}` | Every namespace and collection. Only a cluster key can create or see it. | | `namespace` | `{"kind": "namespace", "namespaces": ["acme"]}` | Every collection in the listed namespaces, also collections created later, and the namespace jobs. More than one namespace needs a cluster key. | | `collection` | `{"kind": "collection", "namespace": "acme", "collections": ["incident-logs"]}` | The named collections in one namespace. | A namespace key can manage namespace and collection webhooks in its own namespace. A collection key can manage collection webhooks for its own collections. Managing a webhook needs `write`. Reading it and its log needs `read`. A webhook does not belong to the key that created it, so a revoked key never breaks it. A research event belongs to the collection that the run searches. A `delete_namespace` job belongs only to its namespace, so a collection webhook does not get it. Deleting a namespace or a collection never deletes a webhook, and the events of the delete still arrive. ## Connect a job to an event *Illustration: Save the job and file IDs from the write response. Match a verified event to those IDs. Use a recovery GET if a notification is missed.* Save the job ID from each accepted write before updating your UI. Match `job_id` or `file_id` from the event to that operation. Use `namespace` and `collection` to route the status to the correct tenant. **Example webhook body** ```json { "events": [ { "id": "evt_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "type": "file.status_changed", "occurred_at": "2026-10-06T12:01:00Z", "cluster": "graphon-engine", "namespace": "acme", "collection": "incident-logs", "job_id": "job_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "file_id": "fil_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "research_id": null, "data": { "status": "ready", "previous_status": "indexing" } } ] } ``` `job.updated` reports operation status. `file.status_changed` reports file readiness. `file.stage_changed` reports each step of an indexing file. `collection.status_changed` reports the collection summary. `research.updated` reports each status of a research run, and `research.progress` reports each new stage. Initial job, file, and research events have `data.previous_status: null`. A new collection in the `empty` state does not emit a creation event. Search and Answer do not emit completion webhooks. A research event names the run in `research_id`, and its `job_id` and `file_id` are null. [Research](/engine/guides/research) shows a batch of research events and how to follow a run. `webhook.test` comes only from Test webhook. Ignore event types that you do not know, and answer `2xx`. ## Follow a file through indexing An `indexing` file can stay in that status for minutes or hours. Its `stage` tells your application which step it is on. The add response and [Get file](/engine/reference/get-file) return the stage. Each change sends `file.stage_changed`, with `data.status`, `data.stage`, and `data.previous_stage`. | Stage | Meaning | | --- | --- | | `engine_queued` | The file waits for an Engine worker. | | `checking_source` | Engine checks the source and sends it for processing. | | `core_queued` | The file waits in the processing queue of its namespace. | | `processing` | Engine reads and processes the content. | | `building_index` | The file is processed and waits for an index that can serve it. | Stages only move forward, in the order of the table. A fast file can skip `processing` or `building_index`. An upload has no stage while it is `pending`, and starts at `core_queued`. A `ready`, `failed`, or `deleting` file has no stage, and that status change sends only `file.status_changed`. **Example stage event** ```json { "events": [ { "id": "evt_01J8Z3K4N5P6Q7R8S9T0V1W2X4", "type": "file.stage_changed", "occurred_at": "2026-10-06T12:01:00Z", "cluster": "graphon-engine", "namespace": "acme", "collection": "incident-logs", "job_id": "job_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "file_id": "fil_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "research_id": null, "data": { "status": "indexing", "stage": "processing", "previous_stage": "core_queued" } } ] } ``` When a file fails, its `file.status_changed` event has `data.error` with `code` and `message`. Save the reason from the event. A failed replacement of a ready file moves to `deleting` at once, so a later GET of that file returns `404`. **Example failed event** ```json { "events": [ { "id": "evt_01J8Z3K4N5P6Q7R8S9T0V1W2X5", "type": "file.status_changed", "occurred_at": "2026-10-06T12:01:00Z", "cluster": "graphon-engine", "namespace": "acme", "collection": "incident-logs", "job_id": "job_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "file_id": "fil_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "research_id": null, "data": { "status": "failed", "previous_status": "indexing", "error": { "code": "source_mismatch", "message": "The object's sha256 is not the declared one." } } } ] } ``` ## Verify before parsing | Header | Type | Meaning | | --- | --- | --- | | `webhook-id` | string | The batch ID, with the `bat_` prefix. Every retry of the batch keeps it. A replay sends its events in a new batch with a new ID. | | `webhook-timestamp` | string containing Unix seconds | The send time of this POST. Refuse a value more than 5 minutes from your clock. | | `webhook-signature` | string | `v1,` and the base64 HMAC-SHA256 signature. During a secret rotation, two values separated by a space, the new secret first. | | `X-Graphon-Events-Webhook` | string | The webhook’s name. One receiver URL can use it to pick the secret. | | `User-Agent` | string | `Graphon-Webhooks/1`. | | `Content-Type` | string | `application/json`. Verify the original body bytes before parsing. | Engine signs each POST with the Standard Webhooks scheme. Verify it with a Standard Webhooks library, such as `standardwebhooks` for Python or Node.js. Give the library the webhook’s secret, the request headers, and the exact raw body bytes. Re-encoding parsed JSON changes the signed bytes, so verify before parsing. **Python · verify one POST** ```python import os from standardwebhooks import Webhook, WebhookVerificationError # The whsec_ secret from the Create webhook answer. webhook = Webhook(os.environ["ENGINE_WEBHOOK_SECRET"]) def handle(raw_body: bytes, headers: dict[str, str]) -> int: # Checks webhook-signature over webhook-id, webhook-timestamp, and the raw # body, and refuses a timestamp more than 5 minutes off. try: batch = webhook.verify(raw_body, headers) except WebhookVerificationError: return 400 for event in batch["events"]: # Oldest first. if already_stored(event["id"]): # A retry or a replay can repeat it. continue store(event) return 204 ``` Without a library, base64-decode the part of the secret after `whsec_` to get the HMAC key. Compute HMAC-SHA256 over `webhook-id`, a period, `webhook-timestamp`, a period, and the raw body. Base64-encode the digest and put `v1,` before it. `webhook-signature` holds one or more such values, separated by spaces. Accept the POST when one value matches in a constant-time compare, and when the timestamp is within 5 minutes of your clock. During a rotation overlap, the header holds two signatures. One match is enough, so a receiver with the previous secret or the new one accepts the POST. When one URL receives several webhooks, pick the secret by `X-Graphon-Events-Webhook`. ## Deduplicate and acknowledge Each POST body is `{"events": [...]}`: the webhook’s oldest queued events, up to 500 events or 1 MiB. Event `id` is the same in every retry, every replay, and at every webhook. A replay sends events in a new batch with a new `webhook-id`, so deduplicate by event `id`, not by `webhook-id`. Store each event ID with the accepted work. If that ID already exists, skip that event without repeating its effects. Accept the whole batch into durable application storage or a queue before returning `2xx`. A `2xx` confirms every event in the batch. Perform slower follow-up work after acceptance. Respond within 30 seconds. Do not set a body size limit that refuses a batch: a `413` is a `4xx`, so every event in that batch fails. Apply your application’s access checks before showing a tenant’s event to a person. A cluster webhook gets the events of every namespace. Route each event by its `namespace` after signature verification. A valid signature identifies a POST from this webhook; it does not sign a human into your application. ## Handle retries | Receiver result | Delivery behavior | | --- | --- | | 2xx | Every event in the batch succeeds. Engine sends the next batch. | | 429, 5xx, a network error, or no answer in 30 seconds | Engine sends the same batch again after 1, 2, 4, 8, 16, 32, and 64 seconds, then every 5 minutes, each ±10%. A longer `Retry-After` replaces the wait, up to 24 hours. | | 3xx | Every event in the batch fails. Engine follows no redirect. Then it sends the next batch. | | 410 | Every event in the batch fails, and Engine turns the webhook off with `disabled_reason` `failing`. | | Other 4xx | Every event in the batch fails. Then Engine sends the next batch. | Engine sends one batch at a time to each webhook, in order. A retrying batch delays only the later events of that webhook. When the receiver answers again, the queued events go out in full batches. Another webhook never waits. A batch retries until the receiver answers or the webhook turns off. After a failed attempt, a webhook older than 72 hours with no `2xx` in the last 72 hours turns off. A `410` turns it off at once. Turning a webhook off ends its queued events `failed`. | `enabled` · `disabled_reason` | Meaning | Next step | | --- | --- | --- | | `true` · `null` | Engine queues and sends each event in the scope and types. | None. | | `false` · `requested` | A key turned it off. Queued events ended `failed`, and new events are not queued. | Update the webhook with `enabled: true`, then replay the gap. | | `false` · `failing` | Engine turned it off: the receiver answered `410`, or sent no `2xx` for 72 hours. | Fix the receiver, check it with Test webhook, enable the webhook, and replay. | ## Recover missed events Each webhook keeps a 30-day delivery log. [List webhook deliveries](/engine/reference/list-webhook-deliveries) shows each event that the webhook got and its last attempt. Filter it by `status`, `event_type`, or `file_id`. [Get webhook delivery](/engine/reference/get-webhook-delivery) adds the event body and every attempt. **GET https://engine.example.com/webhooks/acme-status/deliveries?status=failed** ```http GET /webhooks/acme-status/deliveries?status=failed HTTP/1.1 Host: engine.example.com Authorization: Bearer ``` A failed event is not sent again on its own. [Replay webhook deliveries](/engine/reference/replay-webhook-deliveries) queues past events again, by ID or by time range. `only_failed` picks the events that this webhook already tried and that ended `failed`. A range can reach back before the webhook’s create, to fill a new webhook. Replayed events go out before newer events of the same webhook, so replay a narrow range. **POST https://engine.example.com/webhooks/acme-status:replay** ```http POST /webhooks/acme-status:replay HTTP/1.1 Host: engine.example.com Authorization: Bearer Content-Type: application/json { "since": "2026-10-06T12:00:00Z", "only_failed": true } ``` **202 Accepted** ```json { "queued": 3 } ``` Engine keeps events for 30 days. For an older gap, or for a one-time check, read the resource with the saved job ID. Do not use a GET loop as the live notification path. **GET https://engine.example.com/jobs/job_01J8Z3K4N5P6Q7R8S9T0V1W2X3** ```http GET /jobs/job_01J8Z3K4N5P6Q7R8S9T0V1W2X3 HTTP/1.1 Host: engine.example.com Authorization: Bearer ``` **200 OK** ```json { "id": "job_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "type": "index_file", "status": "succeeded", "namespace": "acme", "collection": "incident-logs", "file_id": "fil_01J8Z3K4N5P6Q7R8S9T0V1W2X3", "error": null, "created_at": "2026-10-06T12:00:00Z", "updated_at": "2026-10-06T12:01:00Z" } ``` A failed job includes `error.code` and `error.message`. Correct the request or source before another write. A `succeeded` job confirms completion. Read the file or collection if your UI needs its latest fields. See [Get job](/engine/reference/get-job), [Cancel job](/engine/reference/cancel-job), and the [events webhook contract](/engine/reference/events). ## Sitemap See the full [sitemap](/sitemap.md) for all pages.