What Data Looks Like for This Category
Data for this category originates from synthesis, fermentation, purification, quality control, and registration support departments at active pharmaceutical ingredient manufacturers. Daily documents include batch production records, material balance records, reaction and purification records, cleaning records, impurity profile reports, and deviation and change documents. Data sources include registered processes, raw material batches, DCS, weighing systems, LIMS, chromatography data systems, and QMS. Update frequency follows per batch and per step. Document structures include multi-step chemical routes, impurity parameters, and batch-related information. Most fields cover professional content such as chemical substance names, yield, and residual solvent percentage. Formulation production scenarios focus on dose uniformity and packaging. The core focus of this category is multi-step chemical routes and impurity profile data.
Constraints Imposed on Document Parsing and Chunking
Documents for this category have scattered sources and mixed formats. Examples include PDFs with embedded chromatograms and Excel batch records with merged cells. Parsing tools must support multi-format content. Professional content such as multi-step reactions and impurity profiles requires retained contextual connections. Chunking must not split process logic or parameter associations. Frequently updated batch documents require efficient parsing processes, to avoid delays to real-time Q&A caused by parsing latency. Professional fields and units in impurity profile reports must be fully preserved. Chunking rules must adapt to structural differences across document types.
Configuration Settings
These configurations apply to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Active pharmaceutical ingredient documents contain multi-step reaction procedures and impurity data. Settings that are too long will lose contextual connections, while settings that are too short will split process logic |
chunkSplitter | Split by paragraph + semantic meaning | Batch production records and reaction records have continuous process logic; splitting by paragraph preserves step integrity |
customPdfParse | Enable embedded image extraction | Impurity profile reports and cleaning records often include embedded chromatograms; text associated with images must be passed to the large model |
UPLOAD_FILE_MAX_SIZE | 500 MB | Adapt to scenarios where multiple documents per batch are uploaded together, and meet storage and parsing requirements for batch registration documents |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Large chromatography data PDF files take longer to parse; this avoids interrupting the parsing process due to timeout |
chunkSettingMode | Manual configuration | Chunking rules must be adjusted for different document types such as impurity profiles and batch records, to adapt to professional content structures |
Three Common Configuration Mistakes
- Symptom: After uploading an impurity profile PDF with embedded chromatograms, the large model cannot access impurity profile data linked to the images. Cause: Embedded image extraction for
customPdfParseis not enabled, so only text content is parsed. - Symptom: After uploading an Excel-format material balance record, table fields in the parsing result are empty or incorrectly formatted. Cause: No dedicated parsing rule is configured for Excel tables. The default parsing logic cannot recognize merged cells or professional fields.
- Symptom: A timeout error occurs when parsing a large PDF document exported from DCS. Cause: The value set for
PARSE_FILE_TIMEOUT_SECONDSis shorter than the actual required parsing duration, leading to task termination.
How to Confirm Proper Configuration
- Upload a single impurity profile PDF with embedded chromatograms. Check if the parsing result includes text descriptions or associated fields for the images.
- Upload an Excel-format material balance record. Check if the parsed table retains core professional fields such as batch number and yield.
- Upload a combined document package of approximately 500 MB. Confirm that the upload and parsing process does not trigger size limit or timeout errors.
- Test adjusting the chunk length. Check if chunks for multi-step reactions retain logical connections between preceding and following procedures.
The values provided here are common starting points, and should be measured against the samples utilized for the specific implementation.