Reconcile Missing Webhook Changes Against Source Records
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
Your webhook receiver is accepting requests, its queue is draining, and its error count is low. Yet a source record has changed while your local application still shows an older value. Receiver health tells you about traffic you observed; it cannot prove that every expected business change reached your application.
To reconcile missing webhook events, compare a bounded set of source records with the corresponding local records using explicit version and field rules. Then separate confirmed application gaps from events still being processed and differences the available evidence cannot explain. Repair only what the evidence supports, and verify the result against the source.
This approach does not require an event-history endpoint. It does require an honest statement of what your source access can establish. A current record can prove that your local state is behind; it usually cannot prove every intermediate change that occurred.
1. Define the completeness claim before collecting records
Start with a narrow operational question: For this account, record type, and comparison boundary, does the local application reflect the source-owned state we can verify? That is more useful than asking whether the webhook system is complete in the abstract.
Distinguish two claims:
- State completeness: the local records match the source state for the fields and population checked.
- Change completeness: every relevant transition was captured and applied, including intermediate states.
A current-state comparison can support the first claim. The second needs stronger evidence, such as a complete change log with documented coverage. If a record changed from open to closed and back to open between checks, matching current values would not reveal the missed transition.
This matters when transitions trigger work. Restoring the current status does not establish whether a required approval, notification, or calculation happened. Keep those questions separate rather than treating a refreshed record as proof that all downstream work is complete.
Write the intended claim into each reconciliation run. A defensible result is “all eligible records in this enumerated scope were compared,” not “no webhooks were lost.”
2. Inventory the source evidence you actually have
Before designing a scan, establish what the source makes available and what each item means. Do not assume a list endpoint, export, version counter, deletion feed, or event replay exists.
- Record lookup: can establish current state for a known identifier, but cannot discover records whose identifiers are unknown locally.
- Record listing or export: may establish a population to compare, subject to its filters, permissions, pagination, and consistency behavior.
- Modification timestamps: may help select candidates, but require documented precision and update semantics.
- Record versions: can establish ordering only if their documented meaning supports it. An opaque revision token supports equality checks, not necessarily greater-than comparisons.
- Local processing evidence: receipt logs, queued work, application outcomes, and stored source versions can explain what your system observed.
Document limitations alongside the evidence. A source response without a field might reflect permissions, field selection, or schema changes rather than a cleared value. A missing record might be deleted, inaccessible, outside the query filter, or temporarily unavailable.
If only individual lookups are available, describe the run as a check of known identifiers. It cannot discover new source records missing entirely from the local system. If the source cannot expose historical transitions, do not claim that the reconciliation reconstructs them.
3. Put boundaries around the scan
A reconciliation job should have a defined population, comparison boundary, resource budget, and stopping rule. Without these, it can become an endless background scan that competes with normal processing and still produces an unclear result.
Record the following before execution:
- The source account or tenant, record type, and eligible filters.
- The source-owned fields under comparison.
- The run start time, comparison cutoff, and allowance for normal processing delay.
- The enumeration method, pagination rules, and restart checkpoint.
- The maximum requests, elapsed time, and permitted repair volume.
- The owner who can approve repairs and resolve ambiguous findings.
Where the source supports a stable snapshot, use its documented boundary. Otherwise, acknowledge that records may change during enumeration. Re-read discrepancies before repair and use an overlapping selection window on later runs where timestamp semantics permit it. Overlap reduces boundary misses but requires deduplication.
A timestamp checkpoint alone is risky when several records share the same timestamp or pages change during retrieval. Use a documented stable cursor or tie-breaker when available. If neither exists, retain the limitation and schedule broader population checks rather than presenting incremental scans as exhaustive.
Respect source request limits. An HTTP 429 response indicates rate limiting, and a Retry-After header may provide a retry delay. Preserve the checkpoint when pausing; do not label an interrupted scan complete.
4. Make comparison rules explicit
Compare only fields with agreed ownership and meaning. A local note or locally calculated score should not be overwritten merely because it differs from the source. For each source-owned field, define normalization rules before examining discrepancies.
Useful rules distinguish missing values from explicit nulls, compare timestamps at a supported precision, and specify whether collection order matters. Avoid broad normalization that hides meaningful differences. Trimming whitespace may be appropriate for one field and destructive for another.
Evaluate comparable source versions together with values:
- Same version, same normalized values: verified match for the selected fields.
- Newer source version, older local version: a candidate application gap; first check pending work and revalidate the source.
- Same version, different values: unexplained mismatch, potentially involving transformation, local mutation, or incorrect version storage.
- Local version newer than the observed source version: possible stale source read, wrong scope, or invalid version interpretation; do not roll back automatically.
Versions alone are not sufficient. A local application can store a new version while failing to apply its fields correctly. Conversely, source-owned values might already match even though local version metadata is stale. Record both comparisons.
Check pending work using evidence, not a blanket waiting period. A matching queued item or active application attempt supports a pending classification. An old timestamp with no processing evidence does not.
5. Use a filled reconciliation worksheet
Fictional worked example: All identifiers, times, versions, counts, and operating thresholds below are hypothetical. This example assumes the source supports record enumeration and documented, monotonically increasing per-record versions. It does not assume event history is available.
The run checks source-owned status and owner fields for support records in one fictional account. Its purpose is current-state reconciliation, not proof that every historical transition was processed.
RUN: REC-HYP-A Account: Fictional account Cedar Record type: Support record Started: 14:10 UTC Candidate cutoff: source updates through 14:00 UTC Normal processing allowance: 10 minutes Scope: records returned by the eligible source listing Comparison: status and owner_id; exact values Version rule: documented per-record increasing integer Budget: 300 source requests; 15 minutes Repair mode: operator approval required Record: CASE-A Source observed: 14:11 UTC Source: version 18; status=closed; owner_id=team-b Local: version 17; status=open; owner_id=team-b Source last modified: 13:42 UTC Processing evidence: no outstanding work for this record; retained receipt/application logs contain no version 18 Recheck: source still version 18 at 14:13 UTC Disposition: CONFIRMED APPLICATION GAP / MISSED CHANGE Action: approve current-state repair for version 18; webhook-loss cause remains unproven Verification: local version 18 and selected values match; source recheck still version 18 Record: CASE-B Source observed: 14:11 UTC Source: version 9; status=assigned; owner_id=team-c Local: version 8; status=new; owner_id=null Source last modified: 13:59 UTC Processing evidence: version 9 received at 14:08 UTC; corresponding work item remains queued Disposition: PENDING EVENT Action: do not launch a competing repair; recheck at 14:18 UTC under the queue service target Verification: after processing, compare version and values; escalate if queued work remains unresolved Record: CASE-C Source observed: 14:12 UTC Source: version 12; status=open; owner_id=team-a Local: version 12; status=closed; owner_id=team-a Processing evidence: version 12 recorded as applied Recheck: source still version 12 with status=open Disposition: UNEXPLAINED MISMATCH Action: retain evidence; investigate transformation, local writes, and version recording before overwrite Verification: unresolved; assigned to integration owner
CASE-A establishes that a source change is missing from the application after the permitted processing interval, with no outstanding work found. It does not establish whether the provider failed to emit a webhook, delivery failed, local evidence expired, or processing silently failed. The repair can proceed without pretending the cause is known.
CASE-B demonstrates why “source newer than local” is not automatically a repair instruction. There is positive evidence that the event is already waiting. CASE-C shows why equal versions cannot substitute for comparing business values.
Add population totals to the worksheet: records enumerated, eligible, compared, matched, pending, confirmed gaps, ambiguous, and unreadable. An unreadable record is not a match. Keep these categories mutually exclusive for reporting, with explanation fields for investigation.
6. Repair confirmed gaps without replaying unknown effects
A repair should use the same validation and ordering protections as normal ingestion, or an explicitly designed equivalent. Avoid an ad hoc database update that bypasses field ownership, audit records, or concurrency checks.
Before applying a repair, fetch fresh source evidence and check the local record again. A delayed event may have resolved the gap since classification. Where supported, condition the write on the local version observed during comparison so a repair cannot overwrite a newer application result.
Separate state repair from side effects. Updating a record to its current source status should not automatically resend a notification or repeat an external action. Historical effects need their own evidence, deduplication checks, and authorization. If missing transitions cannot be reconstructed, report that limitation explicitly.
After repair, verify both the stored version and selected values. If the source has advanced again, classify the record for another comparison rather than calling the earlier repair a stable match.
Keep repair permissions narrow. OWASP authorization guidance recommends least privilege, denying access by default, and checking permissions on every request. A reconciliation worker should only read and repair the accounts and records it is authorized to handle.
7. Recover from partial runs and ambiguous results
Persist progress after completed pages or batches, together with the comparison boundary and run configuration. On restart, reuse the boundary where the source permits it. If a cursor has expired, start a new bounded run and deduplicate findings rather than silently joining incompatible result sets.
Pause repairs when repeated unexplained mismatches suggest a systemic problem. A faulty field mapping can make a large repair batch destructive. Preserve the observed source and local values, their versions, observation times, and pending-work references so an investigator can reproduce the decision.
Use distinct outcomes for incomplete access, failed reads, and actual differences. A permission failure should trigger an access investigation; it should not become a deletion instruction. Apply the same caution to records absent from a listing unless that listing is documented as authoritative for the full scoped population.
Increase scan coverage gradually. Frequent narrow scans reduce delay for recent gaps but may miss old records excluded by timestamp filters. Broader periodic checks cost more reads but can expose those omissions. Choose the balance using observed findings and source-access constraints.
8. Start with one bounded, measurable run
Choose one record type and a small authorized population. Run the comparison without automatic repairs first. Require every discrepancy to receive a disposition and an owner, then approve only repairs whose source evidence and comparison rules are clear.
Measure coverage against the population you can enumerate, not against an assumed total. Track how many eligible records were successfully compared, how long confirmed gaps remain unresolved, how many pending items clear on recheck, and how many ambiguous findings recur.
The next action is concrete: fill in the worksheet header, compare a bounded population, and retain one evidence-backed row for every exception. The run is complete when its scope is accounted for and each exception has a disposition—not merely when the receiver dashboard is green.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications