Replay Webhook Events Without Repeating Completed Actions
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
The processing bug is fixed, but the historical queue contains different realities: events that never ran, events that updated a record before failing, and requests that timed out without a confirmed outcome. Replaying the queue treats those cases as equivalent. They are not.
A safe webhook replay is another controlled attempt at the same business event, not a new event. Preserve original identity, retain completion evidence, and authorise individual effects. Most importantly, a missing completion record does not prove an action never happened.
The objective is a bounded repair: recover demonstrably missing work without repeating completed work or overwriting newer decisions. The filled fictional plan below provides a reusable structure for that decision.
1. Preserve identity and add replay provenance
Keep the original event identity namespaced to its source and workspace. Give each execution its own attempt ID and connect replay attempts to an approved run ID. These identifiers answer different questions:
- Event identity: Which business occurrence is being processed?
- Attempt identity: Which execution produced this result?
- Run identity: Which approved repair included the attempt?
- Effect identity: Which business action must not be repeated?
Design effect keys around the actual uniqueness boundary. If two source events describe the same assignment, separate event-scoped keys might still permit duplicate follow-up tasks. Include the workspace, business object, assignment version, and action type where those define uniqueness. Do not add a processor version merely because the code changed.
Retain the original payload and its hash. Record the processor build, manifest, operator, approval reference, and replay reason separately. If corrected input becomes necessary, preserve it as a traceable derivative with its own hash and transformation record. Never silently replace the stored original.
2. Require evidence that the effect is eligible
A receipt marker proves arrival, not completion. A broad processed flag may conceal partial work. Establish what historical markers mean before using them to select repairs.
Maintain a distinction between event receipt and individual effect outcomes. For every proposed effect, use evidence-backed dispositions:
- Completed: Skip and retain the completion reference.
- Verified not attempted: Consider execution only if current checks and approval also permit it.
- Reconciled absent: Consider execution when an authoritative reconciliation establishes that no matching effect exists.
- In progress: Block concurrent execution and investigate ownership.
- Unknown: Block until reconciliation resolves the uncertainty.
An absent ledger entry is not an eligibility condition. Historical recording may be incomplete. A task might exist even though its completion marker was lost. Positive evidence could be a reliable attempt trace showing failure before the task operation, or a reconciliation against the authoritative task store using the business uniqueness boundary. An incomplete search or a timeout is insufficient.
Evidence also needs concurrency protection. After checking eligibility, atomically claim the effect through controls shared with live processing. Otherwise, a live worker can create the task between the replay's check and write. For local effects, validate the relevant state, create the task, and record completion in one transaction where the storage design supports it.
If historical evidence cannot establish what happened, quarantine the effect. Do not delete deduplication records to force progress.
3. Freeze the selection and inspect current state
Start with a read-only planning pass. Enumerate events, propose effects, and explain exclusions without creating business side effects. Materialise an immutable manifest of event identities and payload hashes rather than approving a changing query such as all failures since the incident.
For each entry, record its previous outcome, evidence references, proposed action, and current-state preconditions. Recheck those conditions at execution because live processing or manual repair may change them after review.
Historical eligibility is not present-day suitability. A contact may have a newer assignment. Restoring an old owner or creating a task for that obsolete assignment could be wrong even when the original task never existed. Specify whether the repair reconstructs history or supplies work that is still relevant now.
The following plan supplies missing work for current assignments only. A newer assignment blocks both the old owner write and its associated task. No fallback action is inferred from a blocked write.
4. Filled replay plan: missing internal follow-up tasks
Fictional worked example: All identifiers, counts, evidence descriptions, outcomes, and thresholds below are hypothetical. This is a recommended operating design, not an Alactic feature description or customer result.
An assignment handler could update the CRM owner and then fail before task creation. A separate validation problem also affected historical events, but its cause has not been established. Fixing task creation does not resolve that validation problem.
The following filled artifact can be reused by replacing each value and attaching the referenced evidence.
- Run and owner: REPLAY-REPAIR-A; integration operator executes; workspace owner approves; processor build repair-A.
- Purpose: Recover missing internal follow-up tasks for assignments that remain current. No outbound communications or reversal of prior effects.
- Frozen selection: Manifest Cedar-A contains 240 original assignment events received from 09:00 to 11:00 UTC on the fictional incident day. Each row includes source identity, workspace, payload hash, assignment version, evidence references, and proposed disposition. Later arrivals are excluded.
- Completed cohort: 120 events have confirmed owner updates and tasks. Include them as skip checks only.
- Missing-task cohort: 70 events have confirmed owner updates. Retained execution traces positively show failure before any task-creation attempt. Each has a linked evidence record; an absent task key alone is not used as proof.
- Validation-failure cohort: 30 events failed validation before any effect. The validation defect is unresolved, and no corrected input or derivative is approved. All 30 remain blocked pending investigation, regardless of the task-creation fix.
- Ambiguous cohort: 20 events have uncertain outbound outcomes and incomplete effect evidence. Quarantine all 20. No automatic release.
- Allowed action: Create one internal follow-up task for a missing-task entry only when positive evidence remains valid, its assignment is still current, no completed or in-progress matching effect exists, and the worker obtains the shared atomic effect claim.
- Explicit exclusions: Unknown and in-progress effects remain blocked even without a completion key. Missing evidence cannot be interpreted as not attempted.
- Forbidden actions: Owner updates, outbound messages, duplicate tasks, source-payload edits, completion-record deletion, compensation, and any task associated with a superseded assignment.
- Validation sample: Inspect 20 manifest entries: five completed, eight missing-task, five validation-failure, and two quarantined entries. Validation-failure entry VF-03 explicitly has a newer assignment than its stored event. The sample is read-only first; only eligible missing-task entries may then execute under approval.
- Sample acceptance: All decisions agree with manual review. Completed tasks remain untouched. All validation failures and quarantined entries produce no effects. VF-03 must be blocked as validation-unresolved and superseded, with no owner update and no associated task creation. Each permitted task has a completion reference.
- Approval boundary: Approval binds the manifest hash, evidence references, processor build, action allowlist, and execution envelope. Adding events, changing effect keys, introducing corrected input, enabling owner writes or messages, or releasing blocked entries requires a revised plan and fresh approval.
- Execution envelope: One replay worker; serial processing per contact; batches of up to 20 eligible entries; reconciliation between batches. Live processing uses the same effect claims.
- Stop conditions: Stop on attempted forbidden action, duplicate effect, workspace or hash mismatch, unexplained reconciliation difference, or newly unknown outcome. Pause affected work on rate limiting.
- Closure evidence: Every manifest entry has a terminal disposition. Each new task has an effect reference. Blocked and quarantined entries remain explicitly unresolved, not repaired.
The validation failures are retained for accountability, not execution. A future repair must explain their cause and, if input changes are needed, identify and approve the exact derivatives before reconsidering those events.
5. Enforce approval in the execution path
The document is not the control. The worker must deny actions outside the approved allowlist. OWASP authorisation guidance recommends least privilege, deny-by-default behaviour, and permission checks on every request. Apply those principles whenever replay work starts or resumes.
Check authority over each event and target object in its workspace. A manifest identifier alone is not authorisation. Revoked or expired approval must prevent new effects, including after restart.
- Verify active approval, manifest identity, and permitted processor build.
- Load the original event and check workspace and payload hash.
- Verify positive eligibility evidence and current assignment state.
- Derive the stable effect key and confirm the proposed action is allowed.
- Acquire the shared atomic claim; block unknown or active claims and skip confirmed completions.
- Recheck mutable preconditions within the protected write boundary, perform the permitted action, and record evidence.
- Classify an uncertain result as unknown rather than automatically retryable.
A replay-only lock cannot protect against live workers. All paths that can produce the same effect need the relevant shared uniqueness controls. An expired claim also does not prove the effect failed: reconcile before taking it over.
Where external effects are involved, use a stable downstream idempotency mechanism if supported and appropriate. Without one, success followed by a lost response creates an uncertainty gap, not exactly-once execution.
6. Work through the sample and reconcile the totals
Hypothetical execution: The read-only sample produces five completed skips, eight eligible task proposals, five validation blocks, and two quarantines. VF-03 records both blocking reasons and produces neither an owner change nor a task. Only the eight eligible proposals execute after approval; each obtains the shared claim and records its new task reference.
Across the remaining missing-task entries, current-state checks find two assignments superseded since planning. Both are blocked without task creation. The final run creates 68 tasks, including the eight sample tasks.
The event reconciliation is therefore: 120 already complete, 68 repaired, two superseded, 30 validation-blocked, and 20 quarantined. These dispositions total 240 manifest entries. The repaired count is 68, not 240 and not 70.
Separately reconcile effects: 68 newly completed tasks, no authorised owner updates, and no authorised outbound messages. Any forbidden-action attempt triggers investigation even if controls prevented it. This distinction exposes a faulty handler that a reassuring processed-event total could conceal.
7. Stop uncertainty from becoming repeated work
If a service accepts a request but its response is lost, seek evidence through supported status lookups, external references, or operational review. If reliable evidence is unavailable, keep the outcome unresolved. Do not turn a timeout into permission to repeat the action.
HTTP 429 indicates rate limiting; a Retry-After header may provide retry guidance, and provider behaviour varies. Pause affected work and preserve event and effect identities when resuming.
Outbound messaging needs a separate approval boundary. Replaying an old event does not renew recipient permission. Under the WhatsApp Business Messaging Policy, honour opt-outs and check current requirements. Approved templates are used to initiate Platform conversations and outside the 24-hour customer-service window, which opens or resets with a user message. Template approval does not guarantee delivery, and automation needs a clear human escalation path. The example avoids these decisions by prohibiting messaging.
8. Resume from evidence, not reset statuses
A stop control should prevent new claims and queue expansion. It may not cancel an in-flight request. Record and reconcile those requests before calling the run cleanly stopped.
Preserve the manifest, attempts, claims, and effect evidence. After investigation, resume only entries that still satisfy approval and eligibility, using original event and effect identities with new attempt records. A change in scope requires renewed approval, not a quiet status reset.
For your next incident, build a read-only manifest and review a mixed sample before execution. Require reviewers to explain the evidence for each allow, skip, and block decision. If they cannot distinguish missing work from missing records, improve the evidence first.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications