What the data for this category looks like
Data sources include registered or archived formulations, raw material safety documents, weighing and emulsification records collected by production equipment, ERP/MES system data, laboratory test reports, and packaging material version archives. Data updates trigger with each batch production or line change operation. A single update covers all process documents for the corresponding batch. Each document includes a list of formulation raw materials, feeding parameters, production step records, test results, and label review information. Fields include raw material name, feeding weight, emulsification temperature, homogenization duration, finished product batch number, and packaging material version number. Numeric and text records from production steps are also included.
Constraints imposed by these characteristics on the vector model and indexing link
Cosmetics manufacturing data is updated frequently with each batch production or line change. Indexes must support incremental synchronization to avoid excessive time spent on full reconstruction, aligning with production workflow pace. Single-batch documents contain multiple field types. Some are numeric parameters such as feeding weight and emulsification temperature. Vector encoding must support mixed-type data. Raw material and label content must comply with archiving requirements. Vector encoding must accurately match compliance fields specified by archiving rules to avoid compliance risks. Exception reports include detailed records of production deviations, with wide variations in text length. Chunking rules must adapt to both long and short texts to prevent critical information from being truncated. Data is linked to batches and production steps. Indexes must retain contextually associated metadata to support subsequent traceability and review.
Configuration Settings
The following configurations are formulated based on the vector indexing function of FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the core paragraph length of a single process document, avoiding truncation of critical formulation and parameter information |
| Overlap Length | 100–150 characters | Retains cross-chunk production step association information, preventing loss of context after chunking |
| Similarity Threshold | 0.75–0.85 | Meets the precise matching requirements for archiving compliance fields, filtering low-relevance non-compliant records |
| Single Search Limit | Top 6 entries | Covers multiple types of documents for a single batch production, avoiding excessive retrieval results that interfere with large language model judgment |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
defaultToken | Calibrate using representative process documents | This is the default token count used for direct text splitting; inspect sample chunks. It does not configure mixed-type data handling. See the v4.17.2 model-field reference |
Three Common Mistakes
- Phenomenon: Retrieval returns the formulation document for the corresponding batch, but the LLM reports no relevant content. Inspect the query, retrieval filters/results, prompt, conversation history, and LLM context budget.
maxTokenis the embedding model input limit, not the total retrieval or LLM context limit. - Phenomenon: Index construction reports an unsupported input format or embedding failure. Inspect document parsing output, sample chunks, the selected embedding model/provider response, and error logs.
defaultTokencontrols the default text-splitting token count; it does not enable mixed-type data handling. - Phenomenon: Incremental index update takes too long, exceeding the response window of the production workflow. Cause: Incremental synchronization rules are not configured, triggering full index reconstruction for each update, with no optimization for batch-level data.
How to Confirm Proper Configuration
- Import the complete process document for a single batch, perform a retrieval test, and verify that the chunk length of returned results conforms to preset rules, with no obvious content truncation.
- Enter a compliance query instruction, such as "archiving-related requirements for a certain raw material", and verify that the retrieval results cover the core fields of the corresponding archiving document.
- Simulate an incremental index update after a single line change, and check whether the update time conforms to the time requirements of the production workflow.
- View index metadata, and confirm that associated information such as batch number and production step has been correctly bound to the corresponding vector entry.
The values provided are common starting points and should be measured against the reader's own samples.