What the data for this category looks like
The data sources for this category include registered process documents, raw material batch ledgers, DCS production control systems, weighing systems, LIMS laboratory information management systems, chromatography data systems, and QMS quality management systems. Documents cover batch production records, material balance sheets, reaction and purification step records, cleaning validation records, impurity profile reports, deviation and change documents. Fields include batch numbers, process step parameters, impurity content values, solvent residue levels, yield data, and more. Data is updated once per production batch and per process step. A single document links multi-stage production and quality data.
What constraints do these characteristics impose on knowledge base retrieval and recall
This category’s data is updated by production batch and process step. A single document links multi-dimensional production and quality parameters, and must meet registration compliance requirements. Retrieval and recall must accurately match specific batch processes, impurity control standards, or change assessment results needed for the tendering scenario. Generalized keyword matching cannot be used as the sole method. Specialized fields such as impurity profiles and solvent residues require the retrieval model to understand specialized industry semantics, to avoid retrieving irrelevant general documents. The high-frequency update feature requires the retrieval pipeline to support quick synchronization of the latest batch data, and to filter outdated old process records.
How to configure the settings
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
similarity | 0.75–0.85 | Match specialized content such as active pharmaceutical ingredient impurity profiles and process parameters, requiring a high similarity threshold to avoid retrieving irrelevant general documents |
limit | Top 10 entries | Tendering documents require coverage of multi-stage data. A single recall provides sufficient associated documents without exceeding the context window |
searchMode | embedding | For semantic associations between specialized fields and process steps, semantic retrieval is more suitable for segmented business scenarios than keyword retrieval |
usingReRank | Enabled | Re-rank the top 10 retrieved results by relevance, ensuring core content such as impurity profiles and yield data appears first |
datasetSearchUsingExtensionQuery | Enabled | Expand the retrieval scope to cover indirectly associated documents including raw material batches and change files, meeting the multi-dimensional data needs of tendering documents |
collectionIds | Collection IDs divided by business department | Limit the retrieval scope to target documents such as batch production records and impurity profile reports, excluding non-tendering-related system logs |
Three common mistakes
- Symptom: Imported Chinese documents including impurity profile reports and batch production records display garbled text, while English content appears normal. Cause: UTF-8 character encoding was not specified during the document upload step, or the source document uses a non-standard encoding format.
- Symptom: Retrieved results include outdated old process records, and valid documents updated after the latest change are not filtered. Cause: Valid documents approved by the QMS were not separately assigned to a collection, or retrieval filtering rules based on update time were not configured.
- Symptom: Knowledge base disk usage exceeds expectations, and it is not possible to split and count the proportions of original files, split chunks, and embedded vectors. Cause: Storage paths for the three types of data were not distinguished according to FastGPT’s storage logic, and old documents from expired batches were not regularly archived.
How to confirm the configuration is correct
- Upload a test impurity profile report document. Use the batch number of this report as the retrieval keyword, and check whether the retrieved results include this document and associated batch production records.
- View the storage path of the knowledge base collection. Confirm that original files, split text chunks, and embedded vectors are stored in their respective directories, and verify that the disk usage statistics logic meets expectations.
- Initiate a retrieval for "a certain batch’s yield and impurity control". Check whether the similarity of the returned results falls within the preset range, and confirm that the re-ranking function is working.
- Upload a test document with non-UTF-8 encoding, check whether garbled text appears after import, and confirm that the character encoding configuration is correct.
The values provided here are common starting points, and should be evaluated against the reader’s own samples.