What the documents look like
Rebate calculation happens after sales close, and its core is policy wording and financial treatment. The main data sources are ERP sales records, dealer remittances, contract ledgers, and the finance system. Day-to-day materials include rebate policies, volume attainment tables, reconciliation statements, and red-letter invoices. These materials update monthly or quarterly, and the same rebate can appear at once in contract clauses, attainment tables, and reconciliation statements, so reconciliation requires aligning entity, period, basis, and amount item by item. Document parsing and chunking exists so that contracts and transaction materials can be cut into fragments that remain searchable and traceable to the original text for later comparison.
Three industry facts shape this step. First, materials differ sharply in structure: a rebate policy carries tiered conditions and exception notes, an attainment table carries figures and basis annotations, and a reconciliation statement carries transaction detail. If chunking separates a condition from the content it governs, the retrieved fragment lacks the premise for judgment. Second, the same rebate may be documented in several materials at once, so chunk boundaries determine whether a comparison can be assembled from retrieved fragments. Third, retrieval is not authorization: a returned fragment is reference content, and the pending draft sentence remains pending human reconciliation.
FastGPT organizes a knowledge base into library, collection, and data. The smallest search unit is the library; collections only classify data and do not affect search results. One data item can map to multiple vectors. Vectors are stored in a PostgresSQL vector field with an HNSW index, while original data is stored in the MongoDB dataset.datas table, whose indexes field records the corresponding vector IDs. During retrieval, vectors are recalled first, then the vector ID is used to fetch original data from MongoDB; when the same original data is hit by multiple routes, results merge and the vector score takes the highest value. Full-text retrieval on PG and similar vector stores uses MongoDB $text with jieba tokenization. Hybrid retrieval merges multiple routes through RRF. ReRank produces a 0-1 score. The reference limit is measured in tokens, and the minimum relevance value takes effect only under semantic retrieval or result reranking. Image retrieval requires image input and matching model capability. Reranking requires a model and must be enabled by configuration. Libraries associated with different vector models cannot be linked together.
Hypothetical paper exercise sample: a brand and a dealer sign a rebate agreement. The material text reads, "Party A grants Party B a 2% rebate on the quarterly remittance amount; returned goods are deducted from the remittance amount; when the attainment rate is below 80%, no rebate applies." A quarterly reconciliation statement reads, "Remittance this quarter is 1,000,000 yuan, of which 50,000 yuan is returns." A volume attainment table reads, "Attainment rate 76%." One draft sentence to be judged reads, "Rebate payable this quarter is 19,000 yuan." Reconciliation compares the draft amount with the original conditions: remittance after deducting returns is 950,000 yuan, and 2% of that is 19,000 yuan, but the attainment rate of 76% is below 80%, and the original condition states that no rebate applies. The draft sentence does not match the original conditions.
Constraints at the document chunking step
Chunk boundaries must keep a condition with the content it governs. In the sample, the clause "when the attainment rate is below 80%, no rebate applies" governs the rebate ratio. If chunking places them in separate fragments, a retrieved fragment can carry the ratio without the disqualifying condition. The comparison then lacks its premise.
Retrieval scope is bounded by the library. Because the smallest search unit is the library, the reconciliation scope is set by which materials are included, and collection grouping does not change search behavior. A fragment returned from outside the included scope cannot support a judgment about the sample.
Index state is distinct from upload state. Original data and retrieval indexes are separate; one data item can carry multiple indexes, and a new index is not created by uploading a file. After material changes, the actual training and index state must be checked before retrieval results can be relied on. The documented guidance for a missing result includes confirming whether data training has completed, lowering the minimum relevance value, raising the reference limit, and checking whether the question is too short or lacks a subject.
Search mode affects which fragments surface. Full-text retrieval suits identifiers, model numbers, proper nouns, and error codes. Semantic retrieval suits natural-language questions and semantically close content. Hybrid retrieval combines both and merges through RRF. Reranking reorders candidate text and yields a 0-1 score. The minimum relevance value is a filter threshold, not a final ordering rule, and after problem optimization, hybrid retrieval, image retrieval, or reranking, results may be merged from multiple routes and need not follow a single vector similarity order.
Image retrieval and reranking are optional paths, not steps executed on every request. Image retrieval requires image input and matching model capability, and whether an image can be retrieved depends on whether a corresponding index was built at ingestion. Reranking requires an available model and must be enabled by configuration.
Configuration and implementation checks
The configuration below centers on parsing and chunking checks for the sample. The numeric starting values are hypothetical starting points for this exercise, used for comparison against manual reconciliation results, and are not official default values.
| Configuration item | Hypothetical starting point for this exercise | Reason for the choice and comparison method |
|---|---|---|
| Knowledge base scope | Include only this rebate agreement and this quarterly reconciliation statement | The smallest search unit is the library; bound the reconciliation scope before searching |
| Chunking method | Chunk clauses and table rows separately | A condition and the content it governs need to share a chunk and avoid separation |
| Index fields | Build a text index for the same data chunk, and add an image description index when needed | One data item can map to multiple vectors; the number of indexes follows content needs |
| Search mode | Hybrid retrieval | Identifier and amount questions suit full-text recall; semantic questions suit vector recall |
| Reference limit | Set the starting point to cover the tokens needed for both materials | Tokens control reference volume and avoid top k instability in a mixed library |
| Minimum relevance | Use the starting point to filter low-relevance fragments | Takes effect only under semantic retrieval or result reranking; compare against manual reconciliation results |
Three common errors
- Splitting a condition from the content it governs. In the sample, if "when the attainment rate is below 80%, no rebate applies" is chunked apart from the rebate ratio, the retrieved fragment lacks the disqualifying condition, and the draft sentence can be misjudged as matching the original text.
- Treating a retrieval result as an authorization or approval outcome. Retrieval returns reference content only; it does not perform authorization, approval, database writes, or external synchronization. The draft sentence in the sample remains pending reconciliation.
- Treating a successful upload as an updated index. After material changes, the actual training and index state must be checked. If training has not completed after the sample materials are updated, retrieval may still return old fragments.
How to verify the setup
- Input the sample original text and the draft sentence, then compare the draft amount against the original conditions. If the retrieved fragment contains both the rebate ratio and the attainment-rate condition, and the comparison result showing the draft sentence does not match the conditions can be manually reviewed, the reconciliation condition is met.
- Input "attainment rate 76%" and "below 80%, no rebate applies" from the sample, then compare whether the retrieved fragment retains both the condition and the value. If the fragment can be traced back to the original text and the condition is not separated, the reconciliation condition is met.
- Input a material not included in this scope and observe whether retrieval returns a fragment from that material. If returned results stay within the included scope, and judgments lacking support are recorded as pending verification, the reconciliation condition is met. An empty result can be checked against downstream design using an explicit paper input; no claim is made that a hypothetical rule necessarily returns an empty result.