What data for this category looks like
This category covers full-process documents for skincare, haircare, makeup, and special cosmetics manufacturing businesses. Documents include formula sheets, weighing and feeding records, emulsification and filling records, cleaning and site clearance records, semi-finished and finished product inspection reports, and label review files. Data sources are registered and filed formulas, raw material safety materials, production equipment logs, ERP/MES systems, laboratory test results, and packaging material version archives. Updates occur in real time as production batches run and lines are changed. Documents include core fields such as raw material names, feeding quantities, process parameters, and filing numbers.
Constraints imposed on vector models and indexing by this category’s characteristics
The multi-source heterogeneous nature, high-frequency update characteristics, and compliance requirements of this category’s data create multiple constraints for the vector model and indexing workflow. Multi-source documents include structured records, semi-structured process files, and unstructured inspection reports. This requires adapting chunking rules for different text structures. Real-time updates triggered by each production batch and line change require indexes to support incremental synchronization. This avoids resource consumption from full reindexing. Compliance matching needs for filed formulas and production data require high accuracy in recall results. Strict controls must be implemented to limit irrelevant data from being included. Wide variation in document lengths exists, with short records and long process sheets present. This requires flexible adjustment of chunking parameters to preserve core information.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to varying document lengths in cosmetics manufacturing scenarios, balances completeness of process sheets and accurate chunking of feeding records |
| Overlap Length | 100–150 characters | Prevents continuous process steps such as emulsification and filling from being split across chunks, maintains information coherence of production workflows |
maxToken | 4096 token | Matches the context limit of most mainstream embedding models, covers total text length of single-batch production records |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
| Similarity Threshold | 0.72–0.85 | Meets high-precision matching requirements for cosmetics formulas and filed data, filters low-correlation production records |
| Single Search Limit | Top 6 results | Balances recall coverage and result readability, avoids excessive redundant information interfering with process review |
Three common mistakes
- Symptom: Index tasks always show no data, and production records cannot be retrieved. Cause: Data sources such as formulas and feeding records are not associated by production batch, resulting in no valid indexable text fragments after chunking.
- Symptom: Search returns results with very low relevance, including irrelevant packaging material versions or historical batch records. Cause: The similarity threshold is set too low, failing to meet the high-precision matching requirements for cosmetics compliance data, and a large amount of irrelevant content is included.
- Symptom: The final set of documents cannot complete the indexing process. Cause: The
maxTokenvalue of a single complete production record exceeds the configured limit, causing chunking failure and preventing valid vector data from being generated.
How to confirm configurations are set properly
- Upload a single complete formula process sheet, check the chunk preview interface, confirm that core process parameters such as emulsification temperature and feeding quantity are not truncated.
- Trigger an incremental indexing task, check the task log in FastGPT v4.17.0, confirm that production records for the corresponding batch have been successfully stored.
- Enter the query "Does the restricted substances for this production meet filing requirements?", retrieve returned results, confirm that raw material safety materials and inspection records for the corresponding batch are linked.
- Adjust the similarity threshold to 0.78, verify whether the number of recalled search results meets the actual needs of production review.
The values provided are common starting points and should be measured against the reader's own samples.