What the documents look like
Rebate calculation happens after sales, and its centre of gravity is policy wording and financial treatment. That separates this category from inventory replenishment material. The documents handled on this page are rebate policies, volume achievement tables, reconciliation statements and red-letter invoices, and they are commonly refreshed on a monthly or quarterly cycle.
Document parsing and chunking has to answer one question for these materials: can the applicable parties, achievement conditions, calculation base and offset method inside a single rebate policy stay together at retrieval time, so that a condition is not separated from the clause it constrains. A rebate policy typically states who the policy applies to, over what period, what threshold must be met, what base the rebate is calculated on, and how returns are treated. If chunking splits the threshold from the base, or the applicable party from the period, the retrieved fragment can no longer tell a reader which figures the rule governs.
Volume achievement tables and reconciliation statements carry status information rather than rules. A reconciliation statement may show a submission date and a review status. Red-letter invoices record an offsetting document. These are distinct from the policy text, and the parsing step needs to preserve which document a fragment came from, together with its version, so that a later reader can trace a fragment back to its source.
Constraints at the document chunking step
In FastGPT a knowledge base consists of libraries, collections and data. The smallest search unit is the library; collections exist to classify data and do not affect search results. One group of data can correspond to several vectors, and the indexes field records the vector IDs for that group. At retrieval time vectors are recalled first, then the vector IDs are used to locate the original data in the dataset.datas table in MongoDB. If several vectors resolve to the same group of original data, the results are merged and the vector score takes the highest value. Chunking therefore has to place a rebate condition, its applicable party and its calculation wording inside the same group of original data. Merging cannot restore a semantic relationship that chunking has already broken.
Full-text search on PG and similar vector stores uses MongoDB $text with jieba tokenisation, and suits exact queries for numbers, models and proper nouns. Semantic search relies on vector distance and suits natural-language questions. Hybrid search merges the two paths through RRF, and Rerank reorders candidate text and produces a 0-1 score. Whether these paths run depends on the actual input and settings. Image retrieval requires an image input and the corresponding model capability, and reranking requires a model and must be enabled in settings.
Hypothetical paper exercise
The following is paper exercise material for this page and points to no real party.
Source text one (rebate policy extract): "This policy applies to East China dealers contracted in the second quarter of 2024. Where quarterly volume achievement reaches no less than 90%, a rebate is accrued at 2% of the actual payment received. Return amounts are deducted from the accrual base."
Source text two (reconciliation statement extract): "Dealer A submitted the second-quarter reconciliation confirmation on 10 July 2024, with status pending review by the dealer's finance team. As of 15 July, brand finance had not received a review receipt."
Draft sentence to be assessed: "Dealer A's second-quarter rebate has been accrued at 2%, and the return amount has been deducted."
Assessment points: source text one supplies the applicable party, the period, the achievement threshold and the deduction rule. Source text two shows the reconciliation confirmation sitting at pending review. The draft sentence asserts that accrual is complete, and there is no evidence of a passed review or of an accrual execution. The draft sentence is unsupported.
Configuration and implementation checks
| Configuration or check item | Setting or action on this page | Relation to the sample |
|---|---|---|
| Search mode | Use hybrid search | The sample contains keywords such as "East China" and "90%" alongside natural-language conditions, and both recall paths can cover them |
| Minimum relevance | Starting value for this hypothetical exercise set at 0.4 | This value takes effect only in semantic search or with reranking, and must be adjusted after comparing against sample retrieval results |
| Reference limit | Set in tokens, starting value for this hypothetical exercise at 3000 | Rebate policy clauses are short, and this must be compared with the length of a single chunked data group |
| Question optimisation | Decide whether to enable it from observed results | The sample question has a clear subject, and enabling it adds one model call and additional latency |
| Chunking check | Manually check whether rebate conditions and applicable parties sit in the same group | The threshold and deduction rule in source text one must fall inside one group of original data |
| Training status check | Check the actual training and index status after upload | Source text two is at pending review, and a successful upload does not substitute for retrieval having been updated |
Three common errors
- Treating "pending review by the dealer's finance team" in source text two as a completed review, and on that basis concluding that the rebate accrual holds. A pending status does not demonstrate approval, and it does not demonstrate a lack of operating permission; it shows that the fact has not yet occurred.
- Treating "return amounts are deducted from the accrual base" in source text one as the same fact as "the return amount has been deducted" in the draft sentence. The first is a rule clause and the second is an execution result. Their subjects and stages differ, and neither supports the other.
- Treating a retrieval hit as authorisation or as business execution. A retrieval returning reference content shows only that the content was recalled. It does not show that reconciliation has been reviewed, that a rebate has been accrued, or that an invoice has been offset.
How to verify the setup
The three checks below use the hypothetical sample above. The outcomes are expected outcomes and were not actually executed.
- Input the draft sentence together with source text two, and compare "has been accrued at 2%" against "pending review by the dealer's finance team". If the retrieval result returns only source text two and shows no record of a passed review or of accrual execution, the draft sentence is unsupported and the evidence gap should be recorded.
- Input "East China second quarter rebate return deduction" and compare whether the recalled content contains the applicable party, the achievement threshold and the deduction rule together. If all three appear within the same group of original data, the chunking satisfies the condition. If they are scattered across different data groups, adjust the chunking and then recheck the training and index status.
- Input a clearly paper-based empty input
[]and compare how the downstream design handles an empty reference. Empty-result handling must be checked against the design, and no claim can be made that a particular fictional rule would necessarily return an empty array on retrieval.
Libraries built on different vector models cannot be associated with each other. Check the library identifier first, exclude those that do not match, and record the actual observation. Do not run a cross-model retrieval in order to compare missed recalls.