Back to all articlesAutomation testing

Rehearse Recovery When a Test Run Leaves the Wrong State

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A test workflow changes the wrong owner, creates duplicate tasks, and sends a message before anyone stops it. The team can identify the faulty rule. What it cannot yet demonstrate is whether it can restore records without overwriting legitimate work—or whether queued actions will recreate the damage.

A recovery rehearsal should establish three things: further effects are contained, only authorised changes are reversed, and everything left behind has a documented disposition. Compare the recovered state with both a known baseline and legitimate changes made afterward. Treat sent communications and completed human actions as consequences to review, not reversible database edits.

The filled worksheet below is a fictional worked example. All identifiers, people, times, counts, and outcomes are hypothetical. Its controls are recommended operating designs, not claims about Alactic capabilities. Check the workspace and current plan for available evidence and repair methods.

1. Define a recovery target before introducing the fault

“Put it back” is too vague to test. Name the affected records, fields that may change, tasks requiring reconciliation, and external effects that cannot be undone.

For an assignment workflow, recovery might mean restoring an incorrect owner while preserving a newer note. It might mean cancelling a duplicate task while leaving its original open. It does not mean deleting everything created after the test began.

  • Containment: the faulty workflow and its pending actions cannot continue affecting the test records.
  • Record repair: eligible fields match the authorised target, with legitimate intervening edits preserved.
  • Task reconciliation: every unexpected task has a disposition and has been checked for further effects.
  • External consequences: messages and completed actions have a reviewer and a documented follow-up decision.
  • Restart: the workflow stays suspended until separately authorised.

These conditions prevent a successful field repair from being mistaken for complete incident closure.

2. Capture evidence that supports selective repair

A screenshot can show a field value without establishing which run changed it. Capture record identifiers, relevant values, existing task identifiers and statuses, communication state, and capture time. Where available, retain change history and version information. Link created objects to the test run rather than relying on timestamps alone.

Use an isolated environment and designated test recipients wherever feasible. If outgoing actions cannot be isolated, rehearse record repair separately and represent external effects with clearly labelled test evidence. Do not contact real recipients merely to make a rehearsal realistic.

The comparison is not simply “baseline versus now.” It is “baseline, failed-run effects, legitimate later changes, and authorised final state.” Without that distinction, a blanket restore can erase valid work.

Decide how to protect repairs from concurrent editing before starting. A fresh read alone does not prevent someone changing a field between that read and the repair write. Use a supported conditional update tied to a version or expected value where available. Otherwise arrange controlled editing during repair, or pause for explicit conflict review. If neither is practical, narrow or stop the rehearsal.

3. Filled worksheet: scope, authority, and starting state

Fictional rehearsal RR-07: a workflow should assign eligible records to a review queue and create one follow-up task only when none exists. The deliberately faulty version assigns the wrong owner, creates duplicate tasks, and attempts communications to designated test recipients.

Worksheet A: control record

  • Scope: isolated test workspace; R-201, R-202, R-203; run RUN-47. No production records are authorised for repair.
  • Baseline: captured at 09:00 in BASE-07, including fields, tasks, and test communication settings.
  • Recovery authoriser: Maya, operations lead, approves the exact repair list.
  • Operator: Ishan, workspace administrator, performs only approved changes.
  • Independent verifier: Lena compares final evidence with the baseline and intervening-change log.
  • Communications reviewer: Jo owns the sent-message review and internal permission follow-up.
  • Stop condition: an unexplained edit, uncertain object attribution, or unconfirmed pending action pauses the affected repair.
  • Repair evidence: retain approval APPROVAL-07, containment CON-07, changes REPAIR-07, reconciliation RECON-07, and final observations.

Knowing an object identifier does not grant permission to modify it. OWASP authorization guidance recommends least privilege, denial by default, and permission checks on every request. Apply that principle to repair tooling by restricting both objects and actions; this is design guidance, not an assessment of a product.

Worksheet B: baseline and failed states

  • R-201 baseline: owner Ana; stage New; T-10 open; no communication queued. Failed state: owner Bot Queue; duplicate T-21 open; communication Q-31 pending.
  • R-202 baseline: owner Ben; stage Contacted; T-11 open; designated test recipient permitted for the test. Failed state: owner Bot Queue; duplicate T-22 open; M-41 recorded as sent, with delivery and receipt unknown.
  • R-203 baseline: owner Ana; stage New; no task; test recipient permission allowed. Failed state: owner Bot Queue; T-23 created. At 09:04, a separate test operator changes permission to opted out, representing a legitimate new request.

R-203 makes the exercise meaningful: copying the baseline back would erase its opt-out. Recovery must preserve that later change.

4. Filled worksheet: contain effects and authorise exact repairs

Stop new effects before repairing old ones. Suspending a workflow does not establish that queued actions, retries, or downstream work have stopped. Inspect those separately.

Worksheet C: ordered repair record

  1. Contain RUN-47. At 09:06, Maya authorises suspension. Ishan confirms Q-31 is cancelled before dispatch and records CON-07 and QUEUE-07. If cancellation were uncertain, Q-31 would remain an unresolved external-action risk.
  2. Approve exact changes. In APPROVAL-07, Maya authorises restoring owners on R-201 to Ana, R-202 to Ben, and R-203 to Ana; cancelling T-21 and T-22; retaining T-23; replacing T-23 instructions with “Internal permission review only; no outbound contact”; and assigning T-23 to Jo. Stage, note, permission, and unrelated task changes are excluded.
  3. Establish edit control. This example assumes no supported conditional-write mechanism has been confirmed. Before rechecking, Maya establishes a controlled editing interval covering affected records and tasks, including other writers. Ishan records the interval and participating operators in REPAIR-07. If editing cannot actually be controlled, repairs pause for conflict review rather than proceeding on a fresh read alone.
  4. Recheck current state. Within that interval, Ishan compares owners and history with the failed-state capture. All three remain Bot Queue without later owner edits. A mismatch stops that repair for a new target decision and approval.
  5. Repair owners selectively. Ishan restores the three approved owners, records each result, and preserves R-203's opt-out. No whole-record snapshot is written back.
  6. Reconcile tasks. T-21 and T-22 are cancelled as erroneous duplicates, retaining their RUN-47 references. T-10 and T-11 stay open. Ishan checks that cancellation has not triggered completion or notification actions.
  7. Apply the approved T-23 changes. Ishan retains it, replaces its instructions with the exact internal-review wording, and assigns it to Jo under APPROVAL-07.
  8. Verify before releasing edit control. Lena checks the affected values and task changes. The team records the end of the controlled interval and investigates any unexpected activity.

Where conditional updates are supported instead, submit the repair against the observed version or expected value. A failed condition is a conflict to review, not permission to retry unconditionally. Neither approach should be assumed available without checking.

5. Filled worksheet: separate reversible state from consequences

Repairing a record does not undo what someone has seen or done. Keep dispatch, delivery, and receipt separate: a sent status alone does not establish delivery or recipient confirmation.

Worksheet D: irreversibility register

  • Q-31: QUEUE-07 confirms cancellation before dispatch. Disposition: cancelled, not sent. This is stronger evidence than a paused campaign.
  • M-41: COM-07 captures the communication record showing sent. No delivery evidence or recipient confirmation is available. Disposition: sent; delivery unknown; receipt unknown. Jo records that it cannot be treated as recalled through a CRM repair. No corrective outbound message is authorised for this controlled test.
  • T-21 and T-22: their status can be changed, but completed work would need separate review. In this hypothetical exercise, the test assignees confirm neither was acted on; their confirmations are retained in RECON-07.
  • T-23: remains open with approved internal-only instructions and Jo as assignee. It is an intentional follow-up, not an unexplained leftover.
  • R-203: remains opted out. Repair must not re-enable contact or create a new outbound action.

In a live incident, do not automatically send an apology because the first message was wrong. A human should review whether further contact is appropriate and permitted.

For WhatsApp, honour permission and opt-outs. Approved templates are used to initiate Platform conversations and outside the 24-hour customer-service window, which opens or resets with a user message. Template approval does not guarantee delivery. Consult the current WhatsApp Business Messaging Policy for precise requirements, and provide a clear human escalation path for automation. Campaign preparation, template review, and actual delivery remain separate states.

6. Verify final state and rehearse an interrupted repair

Verification requires fresh observations, not only the operator's completion notes. Check values, tasks, pending actions, and their connections to the failed run.

Worksheet E: final evidence

  • R-201: Ana; stage New; T-10 open; T-21 cancelled; Q-31 cancelled before dispatch. Evidence: REC-201 and RECON-07.
  • R-202: Ben; stage Contacted; T-11 open; T-22 cancelled. M-41 remains recorded as sent, with delivery and receipt unknown. Evidence: REC-202, RECON-07, COM-07.
  • R-203: Ana; stage New; opted out; T-23 open, assigned to Jo, with approved internal-only instructions. Evidence: REC-203, CHANGE-07, APPROVAL-07.
  • Run-wide: all three created tasks and both communication actions have dispositions. No unexplained RUN-47 objects remain within the inspected scope. The workflow stays suspended.
  • Repeat pass: RECHECK-07 records that no owner repair remains eligible and no task is created, reopened, or communication queued.

Also rehearse interruption: stop after one confirmed repair and resume from its recorded outcome. Do not rerun the whole sequence blindly. Investigate uncertain results before retrying, especially actions that could create or send something twice. Lack of delivery evidence is not confirmation that nothing was sent.

7. Sign off recovery separately from restart

Worksheet F: decision record

  • Verification, 09:25: Lena confirms approved owner and task changes against APPROVAL-07; original tasks and the later opt-out are preserved.
  • Follow-up acceptance: Jo accepts T-23 and records the M-41 disposition: sent, delivery unknown, receipt unknown; no additional test contact authorised.
  • Recovery decision, 09:30: Maya accepts the rehearsal as passed within its stated scope, including the retained task and documented message uncertainty. This is not a claim that all effects returned to baseline.
  • Restart: not authorised. Correct the assignment rule and existing-task check, then run a fresh bounded test before restart review.

The rehearsal fails if it restores visible fields but cannot account for queued actions, overwrites the opt-out, or leaves tasks without a disposition. It also fails if repair requires unapproved authority or uncontrolled overwrites.

Reuse Worksheets A–F for the next bounded test. Track unexplained objects, overwritten legitimate changes, unowned consequences, and time from confirmed containment to verified repair. Choose time targets locally; speed cannot substitute for evidence. The useful outcome is a record another reviewer can follow: what changed, what was repaired, what was deliberately preserved, and who owns the remaining consequences.

Automation testingWorkflow recoveryCRM operationsChange control

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications