What the Data for This Category Looks Like
Multiple business documents from skincare, haircare, makeup, and special cosmetics manufacturers form this category’s data. These documents include formula sheets, weighing and feeding records, emulsification or filling records, cleaning and site clearance records, semi-finished and finished product inspection documents, and label review files. Data sources include registered and archived formulas, raw material safety materials, production equipment, ERP/MES systems, laboratory data, and packaging material version records. Updates trigger per production batch or each line change. Document fields include raw material name, dosage, restricted substance content, batch number, operation time, inspection results, claimed content, and more. Some documents require association with archive numbers and packaging material version information.
What Constraints These Characteristics Impose on the Vector Models and Indexes Workflow
This category’s data characteristics impose multiple constraints on the vector models and indexes workflow. High-frequency batch-level updates require indexes to support incremental synchronization. This avoids repeated processing of full historical data each time a line is changed or a new batch is produced. Documents include compliance fields restricted by filing requirements, such as restricted substance content and archive numbers. Ensure compliance-related content is not split during chunking to maintain complete semantics during retrieval. Multi-source heterogeneous data formats include structured equipment logs, unstructured inspection reports, and label text. Use unified chunking rules to adapt to different document structures while retaining field associations. This prevents loss of critical context during retrieval.
How to Configure Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the semantic integrity of documents such as formula sheets and inspection records, avoiding splitting key compliance fields like restricted substance content and archive numbers |
| Overlap Length | 50–100 characters | Retains context across chunks, ensuring cross-paragraph fields such as batch numbers and operation times are not split |
| maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Similarity Threshold | 0.75–0.85 | Matches the accuracy requirements of compliance reviews, filtering low-relevance non-compliant content |
| Single Search Limit | Top 10 results | Controls the number of retrieval results, avoiding excessive server load in multi-batch data scenarios |
| defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
Three Common Configuration Mistakes
- Phenomenon: When uploading formula sheets or inspection records larger than 10MB, some chunks show abnormal vectorization after chunking. The interface displays errors or empty fields. Cause: Chunking and vectorization parameters are not adjusted for documents with high-frequency batch updates. This leads to too many chunks from large files exceeding the local model’s processing limit.
- Phenomenon: After deploying ollama on a local CPU server and updating FastGPT to versions 4.9 to 4.10, the original indexed knowledge base cannot return retrieval results. Cause: The embedding dimension or index format of the vector model changes after the version upgrade. The vector index for the corresponding knowledge base is not regenerated.
- Phenomenon: Simultaneous upload of multiple batches of production data causes a server crash. After restart, the knowledge base remains in a not ready state, and index creation cannot complete automatically. Cause: Incremental index trigger rules are not configured. The full index task does not automatically resume after server interruption, requiring manual reconstruction.
How to Verify a Correct Configuration
- Upload a single document matching the category characteristics. Check the number of chunks and vectorization status, confirm there are no abnormal prompts.
- Retrieve preset compliance field content. Verify that the relevance of returned results meets business review requirements, adjust corresponding configuration items as needed.
- Trigger an incremental index task. Confirm the knowledge base status updates to ready after the task completes, with no lag or errors.
- Verify index compatibility across versions. Confirm the original knowledge base can be retrieved normally after an upgrade.
The values provided are common starting points and should be measured against the reader's own samples.