What the documents look like
Brand member operations work from CRM systems, enterprise messaging, POS purchase records and mini-program behavior data. The daily documents include member tag tables, outreach plans, coupon rules and privacy authorization records. These materials update daily or weekly and concern customer data and repeat-purchase outreach.
For document comparison, the parties, objects, states and validity scopes that require checking are spread across several document types. A member tag table records tiering results. An outreach plan records outreach arrangements. A coupon rule records the conditions for using a coupon. A privacy authorization record records the scope of authorization. The retrieval and recall step makes these materials retrievable so that a later manual comparison can proceed. Retrieval does not decide whether authorization holds and does not execute any business action.
Constraints at the retrieval step
The knowledge base search node can associate one or more knowledge bases that use the same vector model. Knowledge bases using different vector models cannot be associated together. Check the identifiers first, exclude those that do not match, and record what is actually observed.
Compare semantic, full-text and hybrid search against the test question. No single mode is assumed to suit all materials. The reference cap is measured in tokens, so test with materials of differing length to see whether content stays complete, then adjust the cap. The minimum relevance score takes effect only under semantic search or when result reranking is used. Label relevant and irrelevant samples first, then tune the threshold. Question optimization performs coreference resolution and question expansion. Compare the original question with the optimized question and check whether the subject was altered.
Reference content is output as an array, and its length can be 0. When nothing is found, the output path still runs through. Retrieval results do not constitute an authorization basis, and a similarity score does not perform permission or entity isolation.
Hypothetical drill sample: a brand member operations team maintains an outreach plan document whose original text reads, "From March 1, 2025 to March 31, 2025, send spring coupons to gold members; coupon value 50 yuan; one per person." A separate privacy authorization record reads, "Member A authorized receipt of marketing information on February 20, 2025; authorization scope limited to SMS and enterprise messaging." The draft sentence to be judged is, "Member A may receive spring coupon pushes in March 2025." This draft sentence involves subject, object, period and authorization scope, and must be compared against the two materials above.
Configuration and implementation checks
| Configuration item | Assumed trial starting point | Reason for the choice and comparison method |
|---|---|---|
| Associated knowledge bases | Outreach plan base, privacy authorization record base | Both bases use the same vector model; associate after checking identifiers; exclude cross-model bases |
| Search mode | Hybrid search | The outreach plan contains dates and coupon value; the authorization record contains subject and scope; hybrid search covers both semantics and keywords |
| Reference cap | 800 tokens | 800 tokens is a hypothetical trial starting point. Measure candidate excerpts with the selected model's tokenizer, then compare completeness at 400 and 1200 tokens. Record character counts and token counts separately |
| Minimum relevance score | 0.35 | Assumed starting point; label relevant and irrelevant samples first; compare filtering against 0.2 and 0.5 |
| Question optimization | Enabled | The draft sentence contains references and a time range and needs semantic expansion; compare against the disabled state to check whether the subject is altered |
| Result reranking | Enabled | Second-pass ordering when candidates are numerous; compare ordering stability against the disabled state |
Three common errors
- Treating retrieval results as an authorization basis. Recalling the outreach plan and the privacy authorization record only shows that the content was found. Whether Member A's authorization scope covers spring coupon pushes requires a manual comparison of the scope field in the authorization record against the object field in the outreach plan. Retrieval does not complete that judgment.
- Treating a pending-review state as missing operating permission. In the hypothetical drill sample, if the outreach plan sits in a pending-review state, that only shows the plan has not been approved. It does not show the operator lacks operating permission. Signing, reaching cooperation and contract effectiveness are three distinct facts, and a quotation validity period does not indicate a supply period.
- Forcing association of bases that use different vector models. Bases with different vector models cannot be associated together. Check identifiers first and exclude those that do not match. Do not then infer that association would miss recalls or return redundant content; only record the identifier mismatch actually observed.
How to verify the setup
- Input the outreach plan text and the privacy authorization record text from the hypothetical drill sample, and compare the draft sentence "Member A may receive spring coupon pushes in March 2025" against both originals. Expected observation: retrieval recalls both materials, and the subject, period and authorization scope in the draft sentence match the originals. If the authorization record does not state that spring coupons fall within the authorization scope, mark it as unsupported and preserve the evidence gap.
- Input the draft sentence from the hypothetical drill sample as the retrieval query, and compare whether the recalled content includes the date range and coupon value from the outreach plan. Expected observation: under hybrid search, the date and coupon value are recalled. If full-text search does not match the date, record it as a keyword matching gap and do not infer that the data is missing.
- Input an explicit paper input of []. Check whether the downstream design accepts an empty result. Expected observation: when the reference content array length is 0, the output path still runs through. Do not claim that a fictional rule retrieval necessarily returns []; only check how the empty array is accepted.