Back to all articlesCampaign measurement

Keep campaign outcome coding consistent across reviewers

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

If two reviewers label the same reply differently, a change in campaign results may be a change in judgement rather than recipient behaviour. Before comparing campaigns, have reviewers independently code the same frozen sample, measure their disagreement, and resolve the disputed rules—not just the disputed records.

The practical control is a double-review worksheet that preserves each original label, the reason for disagreement, the adjudicated outcome and the rubric version used. Keep those fields separate. Replacing both reviewers’ answers with the final decision hides the evidence you need to assess reliability.

This article assumes your outcome categories already exist. It addresses how to apply them consistently, not how to define lead qualification or evaluate employee performance. The worked example below is fictional, and every example count, date and threshold is hypothetical.

1. Freeze what reviewers are actually classifying

Many apparent judgement disagreements are really differences in evidence. One reviewer sees a single reply; another reads the whole conversation. One codes the first response; another codes the latest response after a follow-up. Both may be applying their interpretation sensibly to different units.

Write a short coding contract before selecting the sample:

  • Unit: one recipient’s reply thread associated with one campaign, not each individual message.
  • Cutoff: the same fixed observation endpoint for every reviewer.
  • Evidence: the campaign message and the recipient’s replies up to that endpoint, with the same attachments and context available to both reviewers.
  • Output: one primary outcome, plus separate flags where your existing process requires them.
  • Exclusions: records with no recipient reply stay outside this reply-classification exercise and retain their separate reporting status.

This is a recommended design, not a universal unit of measurement. If your reporting uses individual replies instead of threads, keep that unit—but make it explicit and consistent. Do not let reviewers switch between message-level and thread-level decisions within one exercise.

Freeze the evidence as well as the record list. A reply arriving during review should not silently change one reviewer’s task. Put later evidence into the next observation snapshot.

2. Turn existing categories into observable decision rules

A label name is not enough. Reviewers need to know what evidence qualifies, what does not qualify and which rule wins when a reply contains multiple signals.

For the fictional worksheet, assume an existing rubric, version 1.0, with these categories:

  • I — Interested: expresses willingness to take a next step.
  • Q — Question: requests information without a clear next-step commitment.
  • D — Deferred: postpones consideration or contact.
  • N — Negative: declines the offer.
  • O — Opt-out: requests that messages stop.
  • U — Unclear: available evidence does not support a reliable substantive label.

These definitions leave predictable gaps. Does asking for a price list count as a next step? Does positive language outweigh a request to wait? Does a refusal also mean the recipient wants all messages stopped?

Do not settle those questions through private reviewer habits. Add boundaries and precedence rules to the rubric. Where the evidence genuinely cannot distinguish categories, permit an unclear label with a reason rather than forcing a confident guess.

Keep operational actions separate from reporting decisions. In WhatsApp workflows, opt-outs must be honoured; a coding disagreement must not delay that action. The WhatsApp Business Messaging Policy is the relevant source for current messaging requirements. The particular primary-label hierarchy used here is a measurement choice, not a claim that the policy requires this taxonomy.

3. Select a shared sample and protect independent judgement

Use two complementary samples when resources permit. A routine sample drawn from the reporting population helps estimate disagreement in ordinary work. A deliberately selected challenge sample exposes difficult boundaries, such as mixed intent, short acknowledgements or missing context.

Report them separately. A challenge set enriched with ambiguous replies cannot tell you how often reviewers disagree across all campaign replies. A routine sample, meanwhile, may contain too few difficult cases to test important rules.

Give reviewers the same frozen evidence and rubric version. Hide the other reviewer’s label, prior adjudications and campaign performance summaries until both submissions are complete. Those cues can produce agreement without demonstrating that the rubric works independently.

Ask each reviewer for a primary label, the evidence supporting it, the applicable rule and an uncertainty reason where relevant. Confidence alone is not enough: two confident reviewers can apply incompatible rules.

The worksheet below is a small hypothetical challenge sample. It demonstrates the method; it does not establish a population reliability estimate.

4. Filled hypothetical double-review worksheet

Fictional exercise: Sample CAL-01 contains 12 frozen reply threads from an invented campaign. Reviewers A and B independently used rubric 1.0. Both completed their labels before discussion. The final column records adjudication under the clarified rules released as version 1.1.

Record | Reply evidence | A | B | Disagreement reason | Adjudicated label
H01 | Please book the demonstration. | I | I | None | I
H02 | What does the package include? | Q | Q | None | Q
H03 | Send the price list so I can decide. | I | Q | Requested information treated as a committed next step by A | Q
H04 | Ask me again next month. | D | D | None | D
H05 | Not interested. Remove me from these messages. | N | O | Refusal versus explicit stop request; precedence missing | O
H06 | We do not want this offer. | N | N | None | N
H07 | Thanks. | U | U | None; acknowledgement does not establish intent | U
H08 | Sounds useful, but do not contact me until next quarter. | I | D | Positive wording versus explicit postponement | D
H09 | Is support included? | Q | Q | None | Q
H10 | No, that one. | N | U | A inferred a refusal; B identified a missing referent | U
H11 | Stop sending these messages. | O | O | None | O
H12 | Yes, arrange a call. | I | I | None | I

For each disagreement, retain a short adjudication note:

  • H03: requesting decision-making information does not itself establish commitment to the campaign’s proposed next step. Apply Q.
  • H05: an explicit request to stop messages takes primary-label precedence in this rubric. Apply O; preserve the negative-intent evidence in a separate note if needed.
  • H08: explicit postponement takes precedence over general positive language. Apply D.
  • H10: the visible thread does not establish what “that one” refers to. Apply U rather than infer rejection.

The adjudicated label is a documented decision under a rule, not proof of the recipient’s private intent. That distinction matters most for short or incomplete replies.

5. Measure disagreement before discussing it away

Start with exact agreement: the number of records receiving the same independent primary label divided by the number double-reviewed. In this hypothetical worksheet, reviewers agreed on 8 of 12 records, so exact agreement was 66.7% and disagreement was 33.3%.

These are pre-adjudication figures. After adjudication, every record has one final label by design; calling that 100% reviewer agreement would be misleading.

Next, inspect the direction of disagreement. In the worksheet, the four disputed pairs are I/Q, N/O, I/D and N/U. Two involve the boundary of “interested,” but they arise from different causes: information seeking in one case and postponed action in the other.

Also compare the resulting category counts. Reviewer A assigned I to four records; reviewer B assigned I to two. Without calibration, the same replies could therefore produce different apparent interested counts. The worksheet demonstrates a possible measurement distortion, not a campaign performance result.

Record at least these measures for each review round:

  • Number of double-reviewed records and how they were selected.
  • Exact agreement before adjudication.
  • Counts of each disputed label pair.
  • Categories assigned by each reviewer.
  • Number of cases unresolved because evidence is missing.
  • Rubric version and evidence cutoff.

Agreement can look reassuring when one common category dominates the sample. Always inspect category-level disagreements alongside the overall percentage. For larger programmes, a chance-adjusted agreement statistic may add context, but it does not replace examination of the actual decision boundaries.

6. Adjudicate the rule, then publish a versioned update

Have a designated adjudicator compare the evidence with the written rubric. Ask each reviewer to explain the rule used before revealing a preferred answer. Resolve whether the problem was missing evidence, unclear wording, competing precedence rules or a straightforward application error.

If the existing rule is clear, document its application without pretending the rubric changed. If it is incomplete, record a revision. Do not rewrite the rule silently and leave old labels looking as though they were produced under the new standard.

The fictional exercise produces this filled update:

Rule update: CR-014
Previous version: 1.0
New version: 1.1
Hypothetical release date: 14 May 2026
Owner: Campaign measurement lead
Reason: CAL-01 exposed inconsistent boundaries for information requests,
        postponements, stop requests and replies with missing context.

Clarifications:
1. Q applies when information is requested to support a later decision,
   without a clear commitment to the campaign's proposed next step.
2. An explicit postponement takes precedence over general positive wording: D.
3. An explicit request to stop messages takes primary-label precedence: O.
4. Do not infer intent from an unresolved reference; use U and record
   the missing context.

Affected calibration records: H03, H05, H08, H10
Historical treatment: retain original labels and version; assess affected
                     reporting periods before publishing comparisons.
Validation: independently code a fresh sample under version 1.1.

A useful update narrows a decision boundary without becoming a catalogue of individual replies. Include a counterexample: “Please book the demonstration” remains I, while “Send the price list so I can decide” is Q in this fictional rubric. The distinction is commitment, not the presence of a request.

7. Preserve comparability when the rubric changes

A clarification can change reported outcomes even when recipient behaviour stays the same. Choose and disclose how historical data will be treated before presenting a trend across the revision date.

  • Recode affected history: suitable when the evidence remains available and the comparison matters. Apply the revised rule to a defined historical population, not only to records convenient to revisit.
  • Mark a reporting break: suitable when consistent recoding is impractical. Separate periods or annotate the discontinuity instead of presenting an uninterrupted trend.
  • Run a bridge sample: code the same frozen records under both versions to observe which categories move. This helps explain the change but does not automatically provide a valid adjustment for every campaign.

Preserve original reviewer labels, original versions and revised labels as distinct fields. If an incorrect overwrite has already occurred, recover from retained exports or other available records where possible, and identify any unrecoverable uncertainty in the reporting notes.

Avoid repeatedly revising rules to make a preferred performance result appear. Every revision should identify the evidence problem it solves, the categories affected and the validation needed. If a reporting deadline arrives before a material ambiguity is resolved, show the affected records separately rather than assigning convenient outcomes.

8. Validate on fresh cases and set the next review trigger

Re-reviewing the same worksheet tests whether people remember the adjudication. A fresh independently coded sample tests whether the revised rule transfers to new replies.

Set acceptance criteria before seeing that sample. As a hypothetical local operating gate—not an industry benchmark—a team might require at least 90% exact agreement on 30 routine records, no unresolved disagreement involving explicit stop requests, and a written explanation for every remaining disputed pair. A small sample still leaves uncertainty; passing a gate is not proof that all future coding will be consistent.

If the gate is missed, diagnose the failure before scheduling another general calibration meeting. Different evidence calls for a snapshot fix. One unclear boundary calls for a targeted rule revision. Excessive U labels may indicate missing context rather than reviewer caution. Agreement achieved only through discussion calls for another genuinely independent round.

For the next working session, name an owner, freeze a shared sample and have both reviewers submit labels before discussion. Produce three outputs: the pre-adjudication disagreement count, a completed decision log and a dated rubric update if needed. Then schedule a fresh-sample check. Those artifacts make it possible to distinguish a change in recipient response from a change in how your team interprets it.

Campaign measurementOutcome codingData qualityReview workflows

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications