What this category's data looks like
Typical data for this category comes from raw material qualification certificates, ERP/MES systems, weighing equipment, ovens or frying lines, laboratory testing equipment, and food traceability ledgers. Daily documents include formulas and feeding lists, production records, baking or steaming curves, metal detection records, cleaning and disinfection records, and factory inspection reports. Data updates occur per batch, per shift, or daily. Formulas and feeding lists update when production lines change. Production records update per shift. Baking curves generate per batch. Document fields include raw material name, dosage, temperature, time, allergen labeling, test results, and other items. Some documents have clear unit markings.
What constraints these characteristics impose on vector models and indexing
Data updates frequently and production line changes happen often. Indexes must support incremental synchronization to avoid the time cost of full reconstruction. Documents include structured formula fields and semi-structured time-series curves. When splitting chunks, retain field associations to avoid cutting off critical information such as allergen labeling and feeding order. High-risk content such as foreign matter control and allergen labeling requires precise matching. Vector recall relevance standards are higher for this content. Single long documents like complete baking curves have large content lengths. They must adapt to token length limits. Short documents like metal detection records must adapt to basic token configurations. Additionally, associated documents from multiple batches must be recalled simultaneously. The index's recall range must cover multiple recently generated documents.
How to set configurations
These configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapt to the content length of documents such as baking formulas and baking curves, retain complete context of key fields |
| Overlap Length | 100–150 characters | Avoid cutting off cross-chunk associated information such as allergen labeling and feeding order when splitting chunks |
| Similarity Threshold | 0.75–0.85 | Precisely match high-risk content such as foreign matter control and allergen-related queries, filter low-relevance results |
| Single Search Limit | Top 8 entries | Cover associated documents of multiple batches in scenarios with frequent production line changes, meet information needs for process review |
maxToken | 4096 token | Adapt to the content length of long text documents such as complete baking curves, avoid chunk truncation |
defaultToken | 512 token | Adapt to the content length of short documents such as metal detection records and cleaning and disinfection records, optimize index storage |
Three common configuration mistakes
- Phenomenon: Search results have low relevance, and critical information is not recalled. Cause: Insufficient chunk overlap length cuts off the association between allergen labeling and corresponding feeding records, preventing vectors from capturing cross-chunk critical associations.
- Phenomenon: No data exists in the index, or some documents are not indexed. Cause: Incremental update trigger rules are not configured. Only full index tasks are executed, which cannot adapt to the high-frequency update rhythm of per batch and per shift.
- Phenomenon: Search requests cannot return the last set of chunk results, or return empty content. Cause: The
maxTokensetting is smaller than the token count of a single long document, causing long text chunks to fail to load and index correctly.
How to confirm the configuration is correct
- Upload a typical formula and feeding list document, verify that the chunk preview retains complete context of key fields including raw material name, dosage, and allergen labeling.
- Trigger an incremental synchronization task, check if the index update log includes production record documents generated in the last 24 hours.
- Enter a high-risk query such as "This batch's allergen material list", verify that the returned result chunks cover the associated content of the corresponding feeding list.
- Upload a complete baking curve document, confirm that vector data of the corresponding chunks exists in the index.
The configuration values listed above are common starting points, and should be evaluated against sample data specific to the deployment context.