What the documents look like
Tmall flagship store daily operations center on brand authorization, flagship store identity, and Tmall points compensation rules. Materials come from the Tmall merchant backend, the Tmall rule center, ERP order tables, and customer service chat records. Contract and transaction material comparison works with product listing pages, brand authorization letters, delivery commitment records, and after-sales tickets. These materials update daily and per order. Retrieval faces the question of whether the same business object carries consistent parties, periods, and field values across different documents.
A brand authorization letter states the authorizing party, the authorized party, the authorized brand, and the validity period. A delivery commitment record states the order number, the committed delivery time, and the actual delivery time. An after-sales ticket states the ticket number, the linked order, and the handling status. The retrieval step only recalls relevant materials as reference content. It does not perform authorization determination, compensation calculation, or database writes.
Constraints at the retrieval step
The knowledge base search node can associate one or more knowledge bases that use the same vector model. Reference content is output as an array, and its length can be 0. When nothing is found, the output path still runs through. Search modes are semantic retrieval, full-text retrieval, and hybrid retrieval. Hybrid retrieval merges multiple result paths through RRF. Result reranking uses a ReRank model for secondary ordering. The minimum relevance value takes effect only under semantic retrieval or when result reranking is used. The reference limit is measured in tokens. Question optimization handles coreference resolution and question expansion, and it adds one model call before formal retrieval. A similarity score is a relevance measure and does not constitute permission or entity isolation. The same-vector-model requirement applies to associated knowledge bases and does not extend to external systems.
Hypothetical exercise sample: a Tmall flagship store operator prepares to check one brand authorization material against one handover request. One brand authorization letter in the knowledge base reads: "Authorizing party: Certain Brand Management Co., Ltd.; authorized party: Certain E-Commerce Co., Ltd.; authorized brand: Certain; authorization validity period: January 1, 2025 to December 31, 2025." One delivery commitment record in the same knowledge base reads: "Order number: 20250101XXXX; committed delivery time: before 18:00 on January 3, 2025; actual delivery time: 10:00 on January 4, 2025." The draft sentence to be judged reads: "The store completed all order deliveries within the authorization validity period, with no delayed delivery." The handover request to be checked reads: "Please confirm that the above brand authorization material can be used for the delayed delivery determination of January 2025 orders."
Configuration and implementation checks
| Configuration item | Hypothetical starting point for this exercise | Reason for the choice and comparison method |
|---|---|---|
| Associated knowledge bases | Associate only one knowledge base that uses the same vector model | First check the vector model identifier of each base; bases with the same identifier can be associated together, and bases with different identifiers are excluded and recorded |
| Search mode | Run the same test question first with semantic retrieval and then with full-text retrieval | Compare how the two modes recall fields such as authorization period, order number, and delivery time, then decide whether to use hybrid retrieval |
| Reference limit | Set to a token amount that covers one authorization letter and one delivery record | Test with actual materials of different lengths to see whether reference content is complete, and adjust this starting point step by step |
| Minimum relevance | First label relevant samples and irrelevant samples, then take a starting value between 0 and 1 | Takes effect only under semantic retrieval or when result reranking is used; compare whether relevant samples are retained after filtering |
| Question optimization | Turn it off first, then compare with results after turning it on | Check whether the optimized question keeps the authorization party, order number, and period; record when the subject is changed incorrectly |
| Result reranking | When a ReRank model is available, enable it by setting and compare with results when it is not enabled | Reranking produces a score from 0 to 1; compare changes in candidate ordering |
Three common errors
- Treating a retrieval hit as an authorization or compensation determination. The input is the authorization letter text and the draft sentence in the hypothetical sample. Compare whether the recalled reference content and the draft sentence share the same party, the same period, and the same field. The observed phenomenon is that the reference content contains only the authorization period and the delivery time, and the draft sentence's claim of "completed all order deliveries, with no delayed delivery" has no corresponding field support in the materials.
- Treating the minimum relevance value as a permission or entity filter. The input is the same test question and a minimum relevance value between 0 and 1. Compare the reference content returned before and after filtering. The observed phenomenon is that filtering changes only the relevance ordering result, and whether unauthorized records are recalled is not decided by that value.
- Treating a handover request awaiting review as already approved. The input is the handover request in the hypothetical sample. Compare the request text with any approval record. The observed phenomenon is that only the request text exists, with no approving party and no approval time, so the request remains in a pending state.
How to verify the setup
- Input the authorization letter text, the delivery record text, and the draft sentence from the hypothetical sample. Compare whether the party, period, and field in the draft sentence can find corresponding values in the reference content. The observed phenomenon is that the draft sentence goes beyond what the materials record, so it is judged unsupported, and the evidence gap must be retained.
- Input the handover request from the hypothetical sample. Compare the request content with whether an approval record exists in the knowledge base. The observed phenomenon is that only the request text exists and no approval record exists, so the request is pending and does not constitute authorization execution.
- Use an empty array as a paper input to check the downstream design. Compare whether the downstream path still runs through when the reference content output length is 0. The observed phenomenon is that the output path continues to execute, and no claim is made that a fictional rule retrieval necessarily returns an empty array.