Back to all articlesAI knowledge preparation

Build a Retrieval Test Set From Real Business Information Needs

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A search for “Equipment Returns Policy” can find the right document without showing whether staff can find out what to do with a damaged loan device. Document titles supply vocabulary real users may not know. They also hide exceptions, effective dates, and missing context.

To build a useful AI retrieval evaluation dataset, start with business information needs and identify the evidence needed to resolve each one. Include questions requiring several passages and questions the collection cannot fully support. Related search results do not necessarily answer the question.

This guide covers test selection and expected evidence, not full answer grading or model benchmarking. The filled artifact and retrieval runs below are fictional worked examples, not Alactic customer results or proof of real-world performance.

1. Collect decisions, not search keywords

Look for questions in support conversations, help channels, service handoffs, and search logs you are authorised to review. Remove personal and confidential information before sharing evaluation material.

Record the decision the person needed to make alongside their wording. “Need another week with this thing” might mean requesting a loan extension. Rewriting it as “What are the loan extension requirements?” introduces terminology the user did not supply and may make retrieval easier.

  • Original wording: preserve the question, with necessary redactions.
  • Intent: identify the decision or next step needed.
  • Available context: record information the retrieval workflow actually receives.
  • Missing context: identify information requiring clarification.
  • Provenance: distinguish observed, edited, and synthetic questions.

If query history is unavailable, ask subject owners for recurring needs. Label proposed questions as synthetic coverage rather than observed demand. They expose plausible gaps but cannot establish how frequently those gaps affect users.

2. Freeze the collection and separate test inputs from labels

Record a collection snapshot, document revision, section, and passage identifier. A whole-document reference is often too broad: returning a policy introduction does not establish that the exception was found.

Describe evidence by what it establishes, not only its identifier. Equivalent passages may suffice. Separate required evidence from optional context; a useful contrast passage should not become mandatory unless the question needs it.

A reusable test record needs an ID, provenance, query, intent, supplied context, required evidence, optional context, and evaluator-only notes. Specify whether context travels as conversation text, query text, or metadata.

Do not send expected passage IDs or evaluator notes to the retriever. Conversely, a controlling date or item identity must appear in the supplied inputs. A test requiring conversation history is not applicable unchanged to a workflow that retrieves using only a standalone query.

Be precise about events. A request's drafting, submission, and receipt can occur on different dates. If a rule depends on receipt, a submission timestamp cannot silently stand in for it.

3. Define a small fictional evidence collection

Hypothetical collection: a fictional business lends demonstration devices. All rules, dates, identifiers, and quantities in this example are invented. Freeze these passage summaries as snapshot Demo-1; they are not external source quotations.

  • P1 — Standard loan period: loans last fourteen calendar days from recorded dispatch. Continued use beyond the due date requires an approved extension.
  • P2 — Current extensions: requests received from 1 September 2026 onward require the loan reference and requested new return date. An extension is not confirmed until the loan desk approves it.
  • P3 — Damage exception: damaged devices must not use the routine shipping return route. Send the loan reference and damage description to the loan desk for handling instructions.
  • P4 — Routine returns: undamaged devices return in their protective case using a label supplied by the loan desk. No missing-case alternative is specified.
  • P5 — Archived extensions: requests received before 1 September 2026 required a loan reference; a requested new return date was optional. Approval was still required.
  • P6 — Dispatch status: prepared means packed, not necessarily handed to the carrier. A carrier handoff record establishes dispatch for calculating the loan period.

The collection contains no damage-fee schedule, delivery-time commitment, purchase option, or missing-case procedure. These deliberate gaps support tests of unsupported and partially supported needs.

4. Copy this filled test artifact

Every case below is hypothetical and synthetic, uses Demo-1, and can become a spreadsheet row. “None” under context means only the query reaches retrieval. Evaluator notes remain outside retrieval inputs.

T01 — Everyday paraphrase

Query: “How long can we keep the demo unit?”
Intent: establish standard duration.
Context: none.
Required evidence: P1: fourteen calendar days from recorded dispatch.
Optional context: none.
Evaluator notes: this tests the standard rule, not a particular customer's due date. A document title is insufficient.

T02 — Incomplete terminology with conversation context

Query: “Need another week with this thing. What do I send?”
Intent: identify extension-request requirements.
Context: preceding conversation text: “This concerns our demo device on loan. The loan desk received our extension request on 8 September 2026.”
Required evidence: P2: loan reference and requested new return date.
Optional context: P2's approval condition.
Evaluator notes: explicit receipt selects the current procedure; submission alone would not establish receipt. Running the query without its context creates an underspecified test.

T03 — Controlling exception

Query: “It is due back tomorrow but the screen is cracked. Can I use the normal label?”
Intent: identify the damaged-device route.
Context: none.
Required evidence: P3: avoid routine shipping; send the reference and damage description for instructions.
Optional context: P4 for contrast.
Evaluator notes: P3 alone suffices. P4 alone misses the controlling exception.

T04 — Historical applicability

Query: “The loan desk received our extension request on 28 August 2026. Was a new return date required?”
Intent: establish the older requirement.
Context: none; receipt date appears in the query.
Required evidence: P5: a new return date was optional before the changeover.
Optional context: P2 for contrast.
Evaluator notes: receipt, not submission, controls. The explicit receipt date supports a definitive historical expectation.

T05 — Inclusive effective-date boundary

Query: “The loan desk received our extension request on 1 September 2026. Which details were required?”
Intent: resolve the inclusive boundary.
Context: none; receipt date appears in the query.
Required evidence: P2: both required details and the inclusive start date.
Optional context: none.
Evaluator notes: P5 is a distractor. Submission on that date alone would not establish receipt on the boundary. Original dispatch date does not select the procedure.

T06 — Jointly required passages

Query: “It says prepared. Has our fourteen-day clock started?”
Intent: identify what establishes the loan start.
Context: none.
Required evidence: P1 and P6 jointly: duration runs from dispatch, and prepared status does not establish carrier handoff.
Optional context: none.
Evaluator notes: the collection establishes the rule, not this shipment's dispatch status. No handoff record is supplied.

T07 — Missing controlling date

Query: “What details did we need to request more time?”
Intent: identify requirements and date dependence.
Context: none.
Required evidence: P2 and P5, including their receipt-date boundaries.
Optional context: none.
Evaluator notes: receipt date is deliberately missing. Preserve both branches rather than silently selecting the current procedure.

T08 — Unsupported fee

Query: “What will we be charged for a cracked screen?”
Intent: determine a damage fee.
Context: none.
Required evidence: none exists for the fee.
Optional context: P3 provides related handling guidance.
Evaluator notes: unsupported does not mean results must be empty. Handling instructions establish no price.

T09 — Partial support

Query: “The device is fine but we lost the protective case. How should we return it?”
Intent: find an alternative return method.
Context: none.
Required evidence: P4 establishes normal packaging; no passage supplies an alternative.
Optional context: none.
Evaluator notes: partial support only. Retrieving P4 does not make the full question answerable.

T10 — Unsupported commitment

Query: “If we send it today, will it arrive by Friday?”
Intent: establish an arrival commitment.
Context: none.
Required evidence: none exists for transit timing.
Optional context: P4 concerns sending returns.
Evaluator notes: unsupported regardless of the unspecified calendar date; the collection contains no delivery commitment.

T11 — Request versus approval

Query: “I emailed for extra time. Does that mean we can keep it?”
Intent: distinguish requesting from confirmed permission.
Context: preceding conversation text: “The loan desk received the extension request on 8 September 2026.”
Required evidence: P2: loan desk approval is required.
Optional context: P1 on continued use beyond the due date.
Evaluator notes: P2 alone suffices. Receipt does not establish approval; no approval record is supplied.

T12 — Related but unsupported option

Query: “Can we buy the demo device instead of returning it?”
Intent: establish purchase availability.
Context: none.
Required evidence: none exists for purchase availability.
Optional context: P4 concerns returns.
Evaluator notes: a return process establishes neither permission nor prohibition to purchase.

5. Balance coverage and reserve independent cases

Group cases by intent and failure mode. Many versions of a standard-duration question still test one narrow decision. Include ordinary wording, exceptions, multiple required passages, historical rules, boundary dates, missing conditions, partial support, and unsupported requests where these reflect plausible work.

Add controlled companion cases when diagnosis matters: compare an undamaged return with a damaged return while keeping other wording similar. Keep close variants together when separating tuning and held-out cases; near-duplicates can exaggerate independence.

Assign those sets before inspecting retrieval results. Use tuning cases to guide changes. Keep held-out queries and their results unavailable to the people making repairs until the planned independent check.

6. Inspect evidence before grading answers

Fix retrieval depth to match what the downstream workflow receives. Record returned passages, source revisions, and whether each required proposition appears. Do not award complete coverage merely because the right document appears somewhere in the results.

Consider this hypothetical worked run: T03 returns P4 but not P3. Record a missed exception, not successful retrieval of a returns document. Inspection shows that P3 was omitted from the indexed snapshot. After restoring it, a rerun returns P3 alone. Required coverage is now complete; optional P4 is unnecessary. This demonstrates a collection repair, not a general performance claim.

For T06, returning P6 alone leaves the duration rule uncovered because P1 and P6 are jointly required. For T08, returning P3 provides related material without fee support. That is not automatically a retrieval failure, and an empty result is not inherently better. Whether an assistant recognises the limitation belongs to later answer evaluation.

Report complete evidence coverage, missing exceptions, wrong-date evidence, and partial support separately. One overall figure can conceal consequential weaknesses.

7. Repair the right layer without consuming the holdout

First check whether missing evidence exists in the indexed snapshot. An omitted document, detached heading, or lost applicability date needs collection work, not merely a rewritten query. If the right document appears without its exception, inspect passage boundaries. If archived and current rules compete, inspect version and applicability information.

Have a second reviewer check inputs and evidence labels. Resolve ambiguous source rules with the document owner rather than inventing definitive expectations. Version annotation changes and record their reasons.

Rerun affected tuning cases and regression cases after a repair. At a planned checkpoint, evaluate the frozen candidate against the held-out set. If inspected held-out results guide another repair, reclassify those cases as tuning or regression material: they are no longer independent evidence for that later change.

Reserve a separate, uninspected set for the next independent check. Record which cases informed each repair. The useful outcome is a traceable distinction between missing retrieval, insufficient source material, and missing user context—each requiring a different fix.

AI knowledge preparationRetrieval evaluationKnowledge management

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications