What the Data for This Category Looks Like
Cosmetics manufacturing enterprises’ daily documents include formulation process sheets, weighing and feeding records, emulsification or filling records, cleaning and sanitization records, semi-finished and finished product inspection reports, label review documents, and more. Data sources include registered or filed formulations, raw material safety materials, ERP/MES systems, laboratory test data, and packaging material version documents. Data is updated with each production batch or each line change. Document fields include raw material name, dosage, filing number, production batch number, inspection indicators, and others. Some documents include unit parameters such as weight, temperature, and time.
Constraints Imposed on Vector Models and Indexing
Since data is updated frequently according to production batches and line change cycles, the indexing link must support incremental updates to avoid repeated full indexing and improve processing efficiency. Different documents have large structural differences: for example, formulation process sheets are step-by-step text, while inspection reports are tabular data. When splitting chunks, adaptation to the content length of different documents is required to avoid splitting critical business steps. Content such as filed formulations and raw material safety materials has high matching accuracy requirements. Indexing needs to filter low-correlation redundant data to prevent formula or claim errors in bid documents. Field formats of multi-source data are inconsistent. Field mapping and standardization processing must be completed before indexing to ensure consistency of vector embeddings and avoid matching deviations.
Configuration Settings
This configuration applies to FastGPT v4.17.0.
| Configuration Item | Recommended Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the paragraph length of cosmetics manufacturing documents, avoiding splitting continuous formulation process steps |
| Overlap Length | 100–200 characters | Preserves contextual coherence of process steps and raw material lists, preventing critical information from being truncated |
| Similarity Threshold | 0.75–0.85 | Filters low-matching irrelevant documents, avoiding incorrect raw material or filing information in bid documents |
defaultToken | 4096 | Adapts to the total length of batch-imported formulation documents, complying with FastGPT’s basic token limit |
| Single Search Limit | Top 10 entries | Controls the number of recalled documents, avoiding introduction of redundant information when generating bid documents |
| Maximum Context | 8000 characters | Integrates core information such as formulations, raw materials, and filings required for bids, ensuring complete content |
Three Common Mistakes
- The knowledge base remains in the indexing status long after import. This occurs because incremental indexing trigger rules are not configured, or a single imported document (such as a batch of formulation sheets) exceeds the
maxTokenlimit, causing indexing tasks to block. - Abnormally high similarity values after replacing the embedding model. This occurs because the normalization parameters of the embedding model are not uniformly configured, leading to inconsistent vector space scales and results where similarity exceeds reasonable ranges.
- Unable to find an available channel for the m3e model. This occurs because a deployment node for a custom embedding model is not added in system settings, or the node configuration does not point to the correct m3e service address.
How to Verify Proper Configuration
- Upload a single typical document (such as a single-batch formulation process sheet) and check whether the indexing status updates to completed within a reasonable time frame.
- Enter a test query such as "Restriction requirements for a filed raw material" and check whether the recalled documents include filing data for the corresponding category and are sorted in line with business logic.
- Check the embedding model configuration in system settings to confirm that the correct model channel address has been added and there are no error prompts.
- Import batch documents (such as 10 formulation sheets) and verify that indexing tasks can be completed in batches without blocking or failure logs.
The values provided are common starting points and should be measured against the reader's own samples.