What the documents look like
Livestream after-sales work tends to arrive in a burst after a single broadcast. Refund requests, reshipment orders and host commitments cluster together, and the material that retrieval has to work with comes from several systems at once: shop after-sales records, customer service IM logs, warehouse WMS records and logistics traces. The everyday documents are livestream order lists, refund applications, reshipment orders and script templates.
The comparison task is to align subject, object, status and validity scope across those documents, then judge whether a draft reply or a handover item is supported by the material. A refund application that is still pending review does not show that a refund was completed. A buyer request for reshipment does not show that a reshipment was executed. Retrieval returns quoted content; it does not decide authorization, and it does not confirm that a business action occurred.
Constraints at the retrieval step
The knowledge base search node can link one or more knowledge bases that use the same vector model. Knowledge bases built on different vector models cannot be linked together in one node. Check the vector model identifier first, exclude any base that does not match, and record what was actually observed. This same-vector-model requirement applies to the linked knowledge bases only. It does not extend to business registration sites or other external systems.
Semantic retrieval, full-text retrieval and hybrid retrieval are compared against test questions. No single mode is assumed to suit all material. Hybrid retrieval combines multiple result paths through RRF; RRF is a fusion step, not a switch that is turned on. Result reranking requires a rerank model and must be enabled in the settings. The reference limit is measured in tokens. The minimum relevance value takes effect only under semantic retrieval or when result reranking is used. Question optimization performs coreference resolution and question expansion, and it adds one model call. The retrieval output is an array and its length can be zero, so the downstream path still runs when nothing is found. Retrieval and similarity scores do not perform permission checks, entity isolation or business authorization.
Hypothetical paper exercise sample: the source material reads, "Livestream order list A-0715 states: order number 0715-088, item is a summer sun-protection jacket, quantity 1, order time July 15, refund application status is pending review. The customer service IM record shows the buyer raised a reshipment request on July 16. The warehouse WMS does not show this order as shipped out." The draft sentence to be judged is: "Order 0715-088 has completed the refund and a reshipment has been arranged." The comparison points are the same subject, the same period and the same field. A refund status of pending review does not amount to a completed refund. A buyer request does not amount to an executed reshipment.
Configuration and implementation checks
| Configuration item | Hypothetical starting point for this exercise | Reason for the choice and comparison method |
|---|---|---|
| Linked knowledge bases | Link only after-sales material bases that use the same vector model | Check the vector model identifier first, exclude bases that differ, and compare whether the identifiers match |
| Search mode | Semantic retrieval combined with full-text retrieval | Order numbers and status fields suit full-text hits, natural-language questions suit semantic search; compare each against the test questions |
| Reference limit | Set to cover the token length of one complete refund application plus reshipment record | The starting point follows the full length of a single record; compare materials of different lengths to see whether content is cut off |
| Minimum relevance | Label relevant and irrelevant samples first, then set the threshold | This value takes effect only under semantic retrieval or result reranking; compare the filtering outcome across the labelled samples |
| Question optimization | Enable for multi-turn follow-ups and short questions | Compare the original question with the optimized question and check whether the subject was altered |
| Result reranking | Enable when candidates are numerous and the question is complete | A rerank model must be present and the setting must be enabled; compare the order before and after reranking |
Three common errors
- Treating a refund application status of "pending review" as a completed refund. The input is the status field in the source material. The comparison is between "pending review" and "completed". The observable result is that the two values are mutually exclusive, and the draft sentence claiming completion has no support.
- Treating a buyer's reshipment request as an executed reshipment. The input is the customer service IM record. The comparison is between the request being raised and the warehouse shipment record. The observable result is that the WMS does not show a shipment, so the execution fact is not supported.
- Treating a retrieval hit as authorization or a permission decision. The input is the quoted content returned by retrieval. The comparison is between the retrieval result and an authorization status. The observable result is that retrieval produces no authorization conclusion; authorization requires a separate downstream implementation.
How to verify the setup
- Against the sample source material, enter the question "What is the refund application status for order 0715-088?" Compare whether the recalled content contains "pending review". The observable result is that the quoted content matches the status field in the source material.
- Against the draft sentence, enter "Has order 0715-088 completed the refund and been arranged for reshipment?" Compare whether the recalled content shows a completed refund or a shipment record. The observable result is that the material does not state completion or execution, so the claim is judged unsupported.
- Against an empty result, use a paper input of an empty array to check the downstream design. Compare whether the downstream handles the empty reference as designed. The observable result is that the path runs through and produces no fabricated content. For knowledge bases on different vector models, check the identifiers first and exclude the non-matching ones; do not run a cross-model search to compare missed recalls.