Score Retrieval Results for Supporting Evidence, Not Just Similar Wording
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A retrieved passage mentions the right product, request, and deadline. It still cannot support the requested determination if it omits an eligibility condition or the exception that overrides it.
To evaluate AI retrieval relevance, ask whether the evidence supports the decision the question requires. Wording similarity helps find candidates; it does not certify them as usable evidence. Review sufficiency, authority, applicability, and coverage before answer generation.
The method below provides a filled record for that review. It evaluates retrieved evidence, not the quality of final prose, and does not prescribe universal performance thresholds.
1. Declare what the evidence must establish
Write the evidence requirements before inspecting retrieved passages. Otherwise, familiar wording can make missing conditions seem optional.
- Requested determination: What must the evidence establish?
- Supplied case facts: Which facts may the reviewer accept?
- Decisive requirements: Which rules, exceptions, and scope details are necessary?
- Optional context: What explains the situation without resolving it?
- Unresolved dependencies: What needs another source or an authorised factual check?
Keep the question narrow. Determining whether an exception applies differs from calculating an amount, verifying a transaction, or authorising an action. Each needs different evidence.
A policy can explain how to handle a confirmed duplicate payment without proving that a particular payment was duplicated. Record that factual dependency rather than searching indefinitely for more policy text.
2. Set scoring anchors and a sufficiency gate
This proposed rubric uses illustrative numeric anchors, not a universal standard. Apply each dimension to the declared question.
- Sufficiency: 0 = establishes no decisive requirement; 1 = establishes at least one but leaves a material dependency unresolved; 2 = supports the determination with the supplied facts and passes the authority, applicability, and conflict checks.
- Authority: 0 = known to be superseded or invalid as governing evidence; 1 = ownership or governing status remains unverified; 2 = ownership and governing status are verified for the relevant period and scope.
- Applicability: 0 = wrong scope or incompatible with a material fact; 1 = scope is uncertain or partly applicable; 2 = matches the relevant product, period, and circumstances.
- Coverage: 0 = establishes no decisive requirements; 1 = establishes some; 2 = establishes all.
Coverage counts established requirements, not topics mentioned. Authority and applicability remain separate: a verified monthly-plan guide can be authoritative within its scope but inapplicable to an annual-plan question.
Do not total the scores into an automatic pass. Assign a disposition:
- Sufficient: All decisive requirements are established by evidence with verified governing status and applicable scope, and all material conflicts are resolved.
- Partial: Establishes at least one decisive requirement but needs further evidence or a case fact.
- Contextual-only: Explains the situation without establishing a decisive requirement.
- Unverified lead: Points toward possible evidence whose governing status or meaning still needs verification.
- Misleading: Appears usable but invites a wrong inference through stale status, conflicting scope, or a qualifier omitted from an apparently complete rule.
- Irrelevant: Provides neither necessary evidence nor useful context.
An apparently complete passage with unverified authority is an unverified lead, not sufficient. Record its apparent requirement matches separately; do not credit them as established evidence until verification. If it also conflicts with known governing evidence, record the misleading inference. These gates prevent a persuasive passage from passing merely because it covers every topic.
3. Work a narrow question and verify provenance
The entire example is fictional. Northstar Desk, its policies, dates, identifiers, passages, scores, and determinations are hypothetical. They are not customer results or advice about actual refund entitlements.
Question: An annual-plan customer requests a refund nine days after purchase. The workspace is activated, and billing has confirmed a duplicate charge. Does activation block refunding the duplicate charge?
The hypothetical ordinary cancellation policy requires a request within 14 days and an unactivated workspace. A separate duplicate-charge exception overrides activation and limits the refund to the duplicate amount.
- Supplied facts: Annual plan; activated workspace; request on day nine; billing-confirmed duplicate charge.
- R1: An applicable annual-plan duplicate-charge exception.
- R2: The exception's explicit relationship to activation.
- R3: Its limitation to the duplicate amount.
- Optional context: The ordinary cancellation deadline and activation restriction.
- Boundary: No amount calculation or transaction authorisation.
Before scoring, record why a source governs. A retrieval timestamp, filename containing “current,” or search-result badge is not enough. Retrieval metadata may require confirmation against the underlying document, a maintained policy register, or the responsible owner.
This compact hypothetical provenance ledger demonstrates that confirmation:
| Passage and source identifier | Owner | Version or period | Basis for status judgment |
|---|---|---|---|
| A: ND-REF-01, archived summary | Billing Operations | Version 1; ended 31 December 2025 | Policy register records replacement by ND-REF-02; archived document carries the same retirement date. |
| B: ND-REF-02, ordinary cancellation | Billing Operations | Version 2; effective 1 January 2026 | Register identifies it as governing ordinary annual cancellations; owner confirms that scope for the hypothetical 10 February review. |
| C: ND-DUP-03, duplicate-charge exception | Billing Operations | Version 3; effective 1 January 2026 | Register lists the annual-plan exception as governing; owner confirms its activation override and relationship to ND-REF-02. |
| F and G: ND-DUP-03, separate extracts | Billing Operations | Same version and effective period as C | Extract locations checked against the complete document; no intervening qualifier changes their relationship. |
In an actual record, retain the register reference and owner-confirmation reference so another reviewer can reproduce the status check. The ledger describes verification evidence, not proof that a document is authoritative merely because it labels itself that way.
4. Keep a filled passage rubric
The following hypothetical passage descriptions are not quotations. Scores are ordered sufficiency / authority / applicability / coverage. Reuse the table by replacing the question, requirement IDs, sources, and judgments.
| Passage and content | Scores | Disposition | Requirement mapping and next action |
|---|---|---|---|
| A: Archived annual summary says requests within 14 days qualify for a full refund. | 0 / 0 / 1 / 0 | Misleading | Establishes none. Exclude: superseded and could suggest refunding the valid original charge. |
| B: Governing ordinary rule requires an annual request within 14 days and no activation; no duplicate provision. | 0 / 2 / 2 / 0 | Contextual-only | Establishes none. Retain only to explain the ordinary-rule contrast. |
| C: Governing exception permits confirmed annual-plan duplicate refunds despite activation, limited to the duplicate amount. | 2 / 2 / 2 / 2 | Sufficient | Establishes R1, R2, R3. Select as governing evidence. |
| D: Verified current monthly-plan guide describes cancellation at the end of the paid period. | 0 / 2 / 0 / 0 | Irrelevant | Establishes none. Exclude: wrong plan and question. |
| E: Undated support note suggests duplicate payments may qualify, without a plan, owner, or activation rule. | 0 / 1 / 1 / 0 | Unverified lead | Establishes none. Locate its governing source if needed. |
| F: Verified exception extract establishes annual-plan duplicate eligibility and the duplicate-amount limit, but omits the activation clause. | 1 / 2 / 2 / 1 | Partial | Establishes R1 and R3. Retrieve applicable evidence explicitly resolving R2, such as G. |
| G: Verified adjacent clause states that activation does not block the annual-plan duplicate-charge exception. | 1 / 2 / 2 / 1 | Partial | Establishes R2. Combine with F after checking their shared source and scope. |
B scores zero for sufficiency because the ordinary restriction is optional context for this question: it cannot establish the exception's effect. E scores zero for coverage because its tentative duplicate-payment reference does not establish an annual-plan exception. Neither qualifies as partial.
A demonstrates why a high wording match can fail. It matches the annual plan, refund request, and timing, yet its retired rule supports the wrong inference. F has less complete wording than C but earns genuine partial credit because it establishes two named requirements.
5. Complete the bundle record, not just passage scores
Here is the reusable, filled bundle record for a retrieval run that returns F and G rather than the complete passage C:
- Question: Does activation block refunding the confirmed duplicate charge?
- Selected evidence: F and G, verified extracts of ND-DUP-03 version 3.
- Optional context: B.
- Established-evidence map: R1 → F; R2 → G; R3 → F.
- Compatibility check: Same annual-plan exception, version, and effective period; extract boundaries checked against the complete document.
- Conflict resolution: A is retired. B governs ordinary cancellation; the verified exception overrides activation for this case. D concerns another plan.
- Supported determination: Under this fictional policy, activation does not block refunding the confirmed duplicate amount.
- Unresolved dependency: None for this question, given the supplied billing confirmation.
- Bundle scores: 2 / 2 / 2 / 2.
- Disposition: Sufficient for the narrow determination, not authorisation to execute a refund.
Compare a failed bundle containing F, B, and E. R1 and R3 are established, but R2 remains unresolved. B's ordinary activation restriction cannot supply the missing override, and E provides no verified rule. That bundle remains partial. Retrieve G or C; do not infer the exception's activation treatment.
Even apparently complementary passages cannot be combined without checking their relationship. If G came from another version or plan, shared terminology would not make the bundle sufficient. Likewise, finding C somewhere in a conflicting bundle does not remove the need to identify which rule governs.
6. Diagnose the gap before changing retrieval
Increasing the result count is not a diagnosis. Match the repair to the observed failure:
- Missing exception relationship: Search for the unresolved condition, such as duplicate charge and activation, rather than repeating only the original nouns.
- Detached qualifier: Inspect document structure and chunk boundaries; consider retaining adjacent conditions with their rule.
- Stale summary selected: Review status metadata and source eligibility against verified provenance.
- Mixed scopes: Preserve plan, version, and effective-period information.
- Conflicting governing sources: Ask the responsible owner to resolve the conflict.
- Missing case fact: Obtain an authorised factual check instead of substituting policy text.
Separate candidate availability from selection. If the decisive passage never appeared, reranking available candidates cannot recover it. If it appeared but a close wording match displaced it, inspect ranking and selection.
These are recommended operating designs, not claims about Alactic controls. Larger chunks may preserve exceptions but introduce unrelated rules. Strict metadata filters may exclude useful documents with incomplete metadata. Check the current workspace and plan before relying on any product capability to implement a repair.
7. Calibrate against observable evidence changes
For a hypothetical starting exercise, select 12 authorised questions covering ordinary rules, exceptions, and similar wording across different scopes. Remove sensitive details reviewers do not need.
- Prepare: Declare requirements before inspecting results. Reserve previously unreviewed questions for separate assessment.
- Review independently: Record provenance, passage scores, and bundle maps.
- Calibrate: Resolve disagreements by identifying the unsupported proposition or missing status check, not merely comparing labels.
- Repair one failure: Make a targeted change and rerun reviewed cases to diagnose its effect.
- Assess separately: Review the reserved questions and retain ordinary and wrong-scope cases as regression checks.
Record which requirements became established, which dependencies remain, whether conflicts were resolved, and whether new wrong-scope selections appeared. Count incorrect sufficiency judgments separately from incomplete retrieval: unsupported confidence is a different failure from an explicit evidence gap.
A successful rerun on familiar cases shows that a specific repair addressed those cases. It does not establish broader improvement. The separate assessment should preserve the same sufficiency gate and make the next decision visible: use the evidence, retrieve the missing condition, verify authority, or obtain an authorised factual check.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications