What the data for this category looks like
Data for this category comes from daily production documents of bakeries, cookie factories, instant noodle manufacturers, and frozen pastry plants. Documents include formulas and ingredient lists, production records, baking or steaming curves, metal detection logs, cleaning and disinfection logs, and factory release inspection reports. Data sources include raw and auxiliary material qualification certificates, ERP/MES systems, weighing systems, production equipment, laboratory test results, and food traceability ledgers. Updates follow a per-batch, per-shift, or daily schedule. Documents contain fields such as ingredient dosage, baking temperature, time, allergen labels, and cleaning frequency. Units include grams, degrees Celsius, minutes, batch numbers, and others.
What constraints these characteristics impose on vector models and indexing
Documents in this category include sensitive and frequently updated content such as allergen information and production parameters. Indexing must support high-frequency incremental updates to avoid excessive resource usage from full reindexing. Formulas and ingredient lists mostly mix numerical and text fields. Vector models must adapt to encoding logic for mixed fields to ensure semantic consistency of key information. Frequent line changes lead to significant differences between document batches. Indexing must support filtering recall results by batch dimension to narrow the scope of cross-review. Daily or per-shift update schedules require indexing real-time configurations to match production cycles, preventing data delays from reducing review efficiency. Additionally, the direct edible nature of products requires higher accuracy for recall results in cross-review. Irrelevant documents must be limited to avoid interference.
How to set configurations
This page’s configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches natural paragraph breaks for long text sections such as formulas and production curves in this category’s documents, preventing semantic fragmentation |
| Overlap Length | 100–150 characters | Covers key cross-chunk information such as production parameters and allergen labels, preventing important review content from being truncated at chunk boundaries |
defaultToken | 4096 | Adapts to the total token count of bulk ingredient lists and inspection reports in this category’s documents, ensuring complete semantic input |
maxToken | 8192 | Supports complete vector encoding of long batch production records, preventing truncation of critical traceability data |
| Similarity Threshold | 0.75–0.85 | Filters irrelevant documents and accurately recalls cross-review targets with the same formula and production process |
| Single Search Limit | Top 10 results | Controls the scale of search results, adapting to the response needs of rapid review in production scenarios |
Three common errors
- Phenomenon: The knowledge base shows an unindexed status after upload, and a field incompatibility error appears in the vector encoding log. Cause: No adapted preprocessing logic is configured for mixed fields in this category such as ingredient dosage and text descriptions, leading to failure to generate vectors for some fields.
- Phenomenon: The interface prompts that no available channel exists for m3e, making it impossible to select the embedding model for vectorization. Cause: The deployment address and access permissions for the m3e model are not configured according to official documentation, or container network policies restrict model calls.
- Phenomenon: The number of retrieved results does not match expectations, returning only a small number of relevant documents or a large number of irrelevant content. Cause: The similarity threshold or single search limit is not adjusted, leading to overly strict filtering rules or an overly broad search scope.
How to confirm the configuration is properly set
- Upload a typical formula document and check if the system index status shows completed.
- Enter the name of an allergen material as a search term, and verify that the recall results include cleaning and disinfection logs and ingredient lists from the same batch.
- Check the vector model call log to confirm that the
defaultTokenandmaxTokenconfigurations do not trigger truncation errors. - Bulk upload that day’s production records and verify that the index update speed matches the per-shift production cycle.
The values provided are common starting points and should be evaluated against the reader’s own samples.