What this category's production data looks like
Data sources for dairy processing production ledgers include milk source testing, MES systems, homogenization and heat treatment equipment, online temperature and flow collection systems, CIP cleaning systems, and LIMS laboratory information management systems. Data updates occur in real time per milk tank and per batch. Daily documents include raw milk acceptance records, standardized ingredient lists, pasteurization or UHT heat treatment records, CIP cleaning records, filling records, and sample retention and microbial testing reports. Most documents are multi-column structured tables, with fields including batch number, milk collection or testing time, equipment number, physical and chemical or microbial indicators, quantity values, and more. Document fields vary across production links, with no unified fixed format.
What constraints these characteristics impose on knowledge base retrieval and recall
The real-time, high-frequency update rhythm requires the knowledge base to support incremental synchronization, to avoid business delays caused by full updates. Format differences across multi-column structured tables require the retrieval system to support multi-field parsing, rather than only extracting content with a fixed number of columns. The association between a single batch and multiple ledger types requires retrieval to accurately match associated fields such as batch number and tank number, to avoid mixing irrelevant cross-batch data. The perishable nature of liquid raw materials and continuous production requires retrieval results to have high timeliness, prioritizing near-real-time production records. The risk of large-scale recalls associated with errors requires the accuracy of retrieval and recall to meet business compliance requirements.
How to set configurations
These configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
similarity | 0.72–0.80 | Matches the multi-field accuracy requirements of dairy ledgers, avoiding false recalls of irrelevant batch data |
limit | Top 8 entries | Covers multiple types of ledger records for a single batch, while controlling context length |
usingReRank | Enabled | Solves the issue that reranking does not activate when retrieved content is too long, and optimizes sorting of highly relevant documents |
searchMode | mixedRecall | Adapts to field retrieval for multi-column tables, balancing semantic matching and precise keyword matching |
datasetSearchUsingExtensionQuery | Enabled | Expands query term coverage, adapting to exclusive business fields such as raw milk batch numbers and equipment numbers |
collectionIds | Dataset IDs grouped by ledger type | Isolates different types of production records, narrows retrieval scope, and improves recall efficiency |
Three common misconfigurations
- Phenomenon: After uploading a multi-column production ledger Excel file, retrieval results only include the first two columns of fields. Cause: Multi-column data parsing configuration is not enabled, and the system extracts only the first two columns of data by default.
- Phenomenon: When retrieving long-document ledgers, highly relevant documents are ranked at the end of results, and reranking is not triggered. Cause: The
usingReRankreranking function is not enabled, or reranking trigger conditions are set improperly. - Phenomenon: Ledgers returned by retrieval in the workflow include records of irrelevant batches, and target files cannot be accurately located. Cause: Datasets are not grouped by
collectionIds, and retrieval batch range parameters are not limited.
How to confirm configurations are properly set
- Upload a single multi-column production ledger Excel file, check the parsed field list in the knowledge base, and confirm all preset column items are included.
- Initiate a retrieval targeting a specific batch number, review the ranking of returned results, confirm highly relevant documents are ranked at the front, and verify the reranking function is active.
- After configuring
collectionIdsgrouping, switch between different groups to initiate retrieval, and confirm only ledger records of the corresponding category are returned. - Adjust the
similaritythreshold, test recall results under different thresholds, and confirm matching accuracy meets business requirements.
The values provided here are common starting points, and should be evaluated against the reader's own samples.