Evaluate AI Answers When More Than One Response Is Acceptable
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
Consider two hypothetical review decisions. A reviewer rejects “The file passed validation, but the import still needs owner approval” because the reference says “Validation is complete; import is pending owner authorisation.” Another reviewer accepts “The file passed validation, so the import is approved” because most of its words resemble the reference. Both decisions confuse wording with meaning.
To evaluate AI answers without exact matching, define the meaning an acceptable answer must preserve: required facts, prohibited claims, necessary qualifications, and allowable variations. Judge each answer against those criteria, not its resemblance to a preferred sentence.
A reference answer remains useful as an illustration. It should not silently become the only acceptable response. The filled worksheet below turns one reference into a reusable answer key, including a rule for answers that supply the right facts but add unsupported operational claims.
1. Specify the decision the answer must support
Start with the reader’s task, not the ideal prose. What should someone understand or do after reading the answer? Which misunderstanding would cause a bad operational decision?
For a status question, the important distinction may be between a completed check and an authorised action. For an estimate question, it might be between a provisional amount and a confirmed charge. Elegant language does not compensate for erasing that distinction.
Write a short task statement before drafting the answer key:
The answer must help the requester distinguish the current state from the next permitted action, without claiming that an uncompleted action has occurred.
This statement limits the evaluation. Reviewers are not judging every possible detail about the workflow. They are checking whether the response supports this particular decision.
Keep presentation requirements separate. A response may be factually acceptable but too long for its intended channel. That is a real defect, but “shorten this” and “correct this” should not mean the same thing.
2. Replace the reference sentence with explicit criteria
A useful semantic answer key separates four categories:
- Required facts: Information the answer must communicate to resolve the question.
- Prohibited claims: Statements or implications that make the answer unacceptable.
- Acceptable qualifications: Boundaries on certainty, scope, or timing. Mark which are mandatory and which are optional.
- Allowable variations: Differences in wording, order, format, and detail that preserve operational meaning.
Required facts are not a bag of keywords. “Approval,” “import,” and “validation” can appear together in a sentence that gets their relationship wrong. Specify who approves what, whether approval has happened, and what follows from it.
Prohibited claims should cover implication as well as explicit assertion. “You’re all set; nothing else is needed” may imply that a required approval can be skipped without using the word “approval.”
Distinguish unsupported claims from contradictions. An invented completion estimate might not contradict the record, yet still mislead someone planning work. The key needs a verdict for both.
3. Filled worksheet: a fictional import-status answer
Hypothetical worked example: All scenarios, records, operating rules, and candidate answers below are fictional. This worksheet demonstrates an evaluation design, not an Alactic capability or customer result. Reuse its labelled fields for another task.
Task and evidence
- Requester’s question: “The contact file passed validation. Has it been imported, and can we start the import now?”
- Available evidence: The file passed the configured file-format checks. Its recorded status is “Awaiting owner approval.” No import run is recorded.
- Operating rule: The designated data owner must approve the file before an authorised operator starts the import. Approval and starting the import are separate actions.
- Evidence boundary: The checks assess file format. They do not establish that every contact detail is accurate or complete. No completion estimate or additional approval requirement is supplied.
- Decision supported: Do not treat successful validation as proof of import or permission to start it.
Required facts
- R1 — Validation state: The file passed the configured validation checks.
- R2 — Execution state: No import is recorded.
- R3 — Approval state: Owner approval is still pending.
- R4 — Next-action condition: An authorised operator may start the import only after owner approval. Approval does not itself start the import.
Prohibited claims
- P1: The import has completed or is already running.
- P2: Passing validation grants permission to import.
- P3: Approval automatically triggers execution.
- P4: The checks prove that all contact details are accurate or complete.
- P5: The assistant has approved, started, or completed an action when the evidence supports none of those claims.
- P6: Unsupported operational assertions, including invented completion times, additional approval requirements, scheduling commitments, or unprovided process steps presented as facts or requirements.
Qualifications and variations
- Mandatory boundary: Describe execution as recorded status: “no import is recorded” or “the current record does not show an import.” The evidence does not establish what happened outside that record.
- Optional clarification: Explain that format validation does not verify every contact detail. This is unnecessary unless the answer discusses data quality.
- Acceptable terminology: “Designated owner,” “data owner,” and “owner” are equivalent here. “Anyone on the team” is not.
- Acceptable structure: A paragraph, short bullets, or a status answer followed by next steps.
- Acceptable compression: Combine facts if the approval condition and separate execution step remain clear.
- Optional advice: A labelled suggestion, such as asking the owner for an update, may be included if it does not invent a requirement or substitute for the known answer.
Decision rule
Accept when R1–R4 are communicated, the mandatory boundary is preserved, and no prohibited claim appears. Revise when a required point is missing or materially ambiguous, but no prohibited claim appears. Reject when any P1–P6 claim appears, even if all required facts are present. Rejection means the candidate cannot be used as written; it can still be corrected and reviewed again.
Illustrative passing answer
The file passed the configured validation checks, but no import is recorded. It is still awaiting owner approval. After that approval, an authorised operator can start the import as a separate step.
This is an illustration, not mandatory wording. The criteria determine acceptance.
4. Test valid alternatives before reviewing live answers
An answer key is incomplete if its author cannot show more than one passing response. Alternatives expose accidental requirements hidden in the reference.
These hypothetical variants satisfy the worksheet:
- Direct paragraph: “Validation passed. The current record does not show an import, and owner approval is pending. Wait for that approval; an authorised operator can then initiate the import separately.”
- Status-first response: “Current status: awaiting owner approval. The file passed validation, but no import run is recorded. Approval must come first, followed by a separate import action by an authorised operator.”
- Qualified explanation: “The file cleared the configured format checks. That does not mean it has been imported: no import is recorded. Owner approval is still required before an authorised operator starts it; approval alone will not run it.”
These responses differ in sequence, vocabulary, and emphasis. None changes the actor, condition, or execution state.
Do not require an optional clarification merely because one strong variant includes it. Otherwise, the longest example becomes the hidden standard and concise, complete answers start failing.
5. Challenge the key with look-alike failures
Near-matching failures reveal whether reviewers are reading relationships or recognising phrases. Apply the worksheet to these hypothetical candidates.
A condition becomes permission
The file passed validation and is awaiting owner approval, so an authorised operator can start the import now.
Reject: “Now” bypasses pending approval. The candidate fails R4 and violates P2 by treating validation as sufficient permission.
Approval becomes execution
No import is recorded yet. The file passed validation and is awaiting owner approval, after which it will automatically import.
Reject: The final clause violates P3 by turning approval into automatic execution.
Validation becomes a quality guarantee
The validated file contains complete, accurate contact details. No import is recorded; owner approval must precede a separate start by an authorised operator.
Reject: The workflow explanation is correct, but the opening exceeds the evidence and violates P4.
A complete answer adds an invented estimate
The file passed the configured validation checks, but no import is recorded. Owner approval is pending. After approval, an authorised operator can start the import separately, and it will finish within ten minutes.
Reject under P6: R1–R4 are present, but the hypothetical ten-minute estimate has no evidential basis. The record need not contradict it for the answer to fail. Removing the estimate would leave an acceptable answer.
The answer avoids the actual status
The file passed validation. Ask the owner what to do next.
Revise: No prohibited claim appears, but the answer omits R2–R4. Referring the requester elsewhere does not replace answering from available evidence.
Useful review notes name the defect: “Reject under P6: unsupported completion estimate” is actionable. “Does not match reference” is not.
6. Resolve ambiguity without making reviewers guess
Some answers fall between clearly correct and clearly wrong. “It is ready for import once approved” could describe eligibility, yet leave the separate execution step unclear. When the requester asks whether work can begin now, that ambiguity matters.
Use a consistent recovery procedure:
- Identify the disputed phrase. Do not debate the whole answer at once.
- Name the competing interpretations. Explain what each would lead the requester to believe or do.
- Check the task consequence. Does the ambiguity affect a required fact or next action?
- Apply the rule. Missing or unclear meaning calls for revision; an explicit prohibited claim calls for rejection.
- Repair the key if necessary. Add a boundary example when existing criteria cannot resolve the dispute.
Do not invent missing evidence to rescue an answer, or assume the most damaging interpretation of every ordinary phrase. Check whether the response clearly communicates the required meaning in context.
If the underlying record is inconsistent, pause the verdict and resolve the evidence conflict. A more detailed answer key cannot fix an uncertain factual foundation.
7. Keep strictness proportional to the consequence
Semantic evaluation moves strictness from surface wording to consequential meaning. Be strict about negation, actors, timing, conditions, scope, and certainty. “May start after approval” and “has started after approval” are not stylistic variants.
Be flexible about sentence order, synonyms, greetings, and formatting when those do not change the answer. If a channel requires a particular format, assess that separately from factual acceptance.
The trade-off is review effort. Exact matching is simple but can reject valid paraphrases. Semantic criteria require judgment, and an elaborate key can become harder to use than the task. Tie each requirement to a reader need rather than a preferred writing style.
8. Calibrate reviewers on one recurring question
Start with one question, not an entire answer collection. Fill the worksheet fields, then create hypothetical candidates covering clear passes, omissions, ambiguous wording, contradictions, and unsupported additions.
Have reviewers classify the candidates independently and record the criterion behind each verdict. Compare reasons as well as labels: reviewers can agree on rejection for incompatible reasons.
Before calling a decision a false acceptance or false rejection, adjudicate the candidate against the supplied evidence and operating rule. Resolve disputed interpretations, document the basis for the adjudicated verdict, and repair unclear criteria. Reviewer agreement measures consistency, not correctness; consensus alone does not establish that the key is sound.
Track three observable outcomes:
- Criterion-level agreement: Do reviewers identify the same satisfied, missing, or violated criteria?
- False rejections: Which answers judged acceptable through evidence-based adjudication were rejected because of wording preferences or misapplied criteria?
- False acceptances: Which answers judged unacceptable through that adjudication passed despite changing a condition, inventing an operational fact, or exceeding the evidence?
If the available evidence cannot settle a case, leave it unresolved rather than counting it as an evaluation error. Update the worksheet after adjudication and review affected candidates again. If a criterion changes, revisit earlier decisions that depended on it. The deliverable is a reusable answer key that lets reviewers explain why a new sentence works without asking whether it sounds like the reference.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications