What This Category of Data Looks Like
This category of data primarily originates from approved formulation processes, MES electronic batch records, LIMS systems, and quality management systems. It covers production, quality assurance, quality control, and warehousing departments. Daily documents include master production documents, batch production records, cleaning validation records, deviation/CAPA/change documents, and batch release review forms. Data is updated per production batch, each critical process step, and monitoring plan. Fields include material batch numbers, process parameter values, deviation descriptions, test results, and other relevant fields, with standardized units attached.
Constraints for Vector Models and Indexing
This category of data is updated frequently by production batch and process step, and must be synchronized with production operations. Indexes must support incremental updates to avoid re-indexing full datasets. Documents include structured batch records and unstructured deviation reports, with large differences in field lengths across document types. Chunking rules must be adapted to preserve complete process parameters and exception descriptions. GMP requirements mandate full data traceability across the production lifecycle. Indexes must retain the association between original data and production batches. They must also ensure retrieval results meet GMP document compliance requirements, and exclude non-compliant data.
Configuration Settings
| Configuration Item | Recommended Setting Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the single-section content length of batch production records and deviation reports, avoids splitting critical process parameters |
| Overlap Length | 50–100 characters | Preserves contextual connections between adjacent chunks, ensures continuous retrieval of process steps |
| Similarity Threshold | 0.75–0.85 | Meets the precision requirements for GMP-compliant retrieval, filters low-correlation non-compliant documents |
defaultToken | 1000 token | Adapts to the average length of a single process document, ensures content completeness for basic retrieval |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Single Search Limit | Top 8 results | Controls the number of retrieval results, avoids introducing excessive unnecessary historical batch data |
Three Common Misconfigurations
- Issue: After upgrading to FastGPT v4.17.0, the same CSV-formatted batch record file fails to chunk correctly. Cause: The new version has stricter character encoding validation for chunking rules, and does not support files exported in older versions with non-UTF-8 encoding.
- Issue: The knowledge base indexing gets stuck, with the interface showing incomplete indexing status for more than 30 minutes. Cause: Incremental index trigger logic is not configured, and full indexing of frequently updated batch record data exceeds processing time limits.
- Issue: Retrieval results include batch records that have not completed release review. Cause: The similarity threshold is set too low, and the filtering rules do not associate with the batch release status field, leading to the inclusion of non-compliant data.
How to Confirm Proper Configuration
- Upload a single complete batch production record, check whether all critical process parameters and batch identifiers are preserved in the chunked content.
- Trigger an incremental indexing operation, verify that only newly added batch data is included in the index, and that historical data is not reprocessed.
- Enter search terms related to exception report drafting, check whether the similarity of returned results falls within the preset range.
- Review indexing operation logs, confirm that there are no encoding errors, timeout errors, or data association anomalies.
The configuration values provided are common starting points, and should be evaluated against samples relevant to the specific operational use case.