What the documents look like
A Shopify standalone store has the merchant configure payment, shipping and fulfillment services directly, rather than receiving a platform-managed returns process. Order details, shipping rate templates, shipping labels and refund records therefore sit across Shopify Admin, payment gateways, carrier systems and inventory location data. Fulfillment status updates per order and per day.
For operations handover, these materials frequently rest in different authorization states. A refund record may be only a request submitted by a support agent. A shipping label may have been purchased but not yet scanned by the carrier. A shipping rate template may still be a draft. Knowledge base retrieval and recall can return existing documents as reference content. It cannot treat a retrieval hit as proof that authorization was granted or that a business action was completed.
Constraints at the retrieval step
The knowledge base search node can associate one or more knowledge bases that use the same vector model. Its reference output is an array whose length can be 0, and the output path still completes when nothing is retrieved. Semantic retrieval, full-text retrieval and hybrid retrieval are selected according to the question. Hybrid retrieval merges multiple result paths through RRF. A ReRank model performs a second ordering of candidates and returns a score from 0 to 1. The minimum relevance value takes effect only under semantic retrieval or when result reordering is used. The reference limit is measured in tokens. Question optimization handles coreference resolution and question expansion. These capabilities act on content relevance. They do not determine whether a material has been authorized or whether an action has been executed.
Hypothetical paper exercise sample: one handover record in the knowledge base reads "On 3 June 2025, operator A submitted the refund request for order #8801 to finance for review, with status pending review; the shipping label for the same order was purchased on 2 June, and the carrier system shows no pickup record." A draft sentence to be assessed reads "Order #8801 refund completed, shipment handed over to carrier." During checking, retrieve that record and compare the subject, object and status in the draft sentence against the original text. The original states only that the refund is pending review and the label is purchased without pickup. The draft sentence states that the refund is completed and the shipment has been handed to the carrier. The two do not agree, so the draft sentence cannot serve as evidence that handover is complete.
Configuration and implementation checks
| Configuration item | Hypothetical starting point for this exercise | Reason and comparison method |
|---|---|---|
| Associated knowledge bases | Associate only fulfillment document bases that use the same vector model | Check the vector model identifier of each base; exclude any base whose identifier differs |
| Search mode | Run the same set of handover questions through semantic retrieval and full-text retrieval separately | Compare whether the retrieved content from each mode covers order numbers and status terms, then decide whether to use hybrid retrieval |
| Reference limit | Set to a token amount that can hold one complete handover record | Compare whether reference content is truncated using handover records of different lengths, and adjust step by step |
| Minimum relevance | Annotate relevant and irrelevant samples first, then take an initial threshold | Takes effect only under semantic retrieval or with reordering enabled; compare whether relevant records are wrongly removed after filtering |
| Question optimization | Enabled for multi-turn follow-up scenarios, disabled for single-turn explicit questions | Compare the original question with the optimized question, and check whether order numbers or status terms were altered |
| Result reordering | Enabled when there are many candidates and a reordering model is configured | Compare ordering before and after reordering, and check the 0-1 score against manual annotation |
Three common errors
- Treating a retrieval hit as completed authorization. The input is the draft sentence "Order #8801 refund completed." The comparison object is the original text "status pending review." The observable result is that the original does not state completion, so the retrieval hit does not constitute evidence that the refund was completed or that authorization was executed.
- Associating knowledge bases with different vector models. The input is the vector model identifier of two bases. Compare whether the two identifiers match. The observable result is that a base with a differing identifier should be excluded, and cross-model retrieval should not be run to compare missed recall.
- Confusing the reference limit with minimum relevance. The input is one long handover record. Compare the token amount of the reference limit against the minimum relevance threshold. The observable result is that the reference limit controls the length of reference content while minimum relevance controls filtering, and the two cannot substitute for each other.
How to verify the setup
- Run retrieval using the hypothetical sample original text and the draft sentence. The input is the order #8801 handover record and the draft sentence. Compare the status terms in the draft sentence against the status terms in the original text. The observable result is that when the draft sentence and the original disagree, the retrieval result must not be used to conclude that the refund is completed or that shipment was handed to the carrier.
- Check the downstream design with a paper input of an empty reference array []. The input is the situation where the node output reference content has length 0. Compare whether the downstream path still completes. The observable result is that the downstream can accept empty references, and no claim is made that a fictional rule will always return [] on retrieval.
- Check the vector model identifiers of the knowledge bases to be associated. The input is the model identifier of each base. Compare whether the identifiers match. The observable result is that only bases with matching identifiers may be associated together; bases with differing identifiers are excluded and the actual observation is recorded, without inferring the runtime result of cross-model retrieval.