What the data for this category looks like
The core data for this category comes from cell banks and seed lots, process analysis, cleanroom monitoring, MES, LIMS, and equipment integrity testing. Daily documents include cell bank or seed lot records, culture and purification records, aseptic filling records, environmental monitoring reports, media fill simulation records, sterility test reports, and batch release documents. Data update cycles include batch archive documents generated per batch, real-time uploaded cleanroom monitoring data, and environmental records generated per fixed monitoring plans. Documents include fields such as batch number, culture temperature, sedimentary bacteria concentration, filling time, with corresponding professional units including degrees Celsius, CFU/cubic meter, hours, and others.
What constraints these characteristics impose on the document parsing and chunking link
This category has diverse data sources. These include structured LIMS spreadsheet documents, as well as unstructured process records and test reports. This requires the parsing link to support both structured field extraction and non-standard text chunking. Data update cycles vary greatly. Real-time monitoring data requires fast parsing. Batch archive documents may contain multi-page long content, requiring flexible adjustment of chunking parameters. Professional fields and units in documents must be fully retained to avoid loss of key process parameters after parsing. In addition, batch data has strong connectivity. Chunking must retain contextual logic to facilitate association of records from different links of the same batch during subsequent traceability.
How to set the configurations
These configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Biologics documents contain long process descriptions and structured data. This range can retain the complete context of a single batch record and avoid splitting key parameters |
chunkSplitter | Split by paragraph + heading | Adapts to multi-type document structures, retains the hierarchical logic of records, and facilitates association of data from different links of the same batch during subsequent traceability |
customPdfParse | Enable structured extraction mode | Adapts to PDF spreadsheet documents exported from LIMS, and can accurately extract structured fields such as batch number and culture parameters |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Large batch release documents or multi-page cleanroom monitoring reports take a long time to parse. This duration covers the complete parsing process |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports uploading complete batch archive documents, avoiding parsing failure due to oversized files |
chunkSettingMode | Manual configuration | Allows adjustment of chunking parameters for different document types such as record-type and monitoring-type documents, to adapt to the diverse document formats of this category |
Three common misconfigurations
- Symptom: File upload parsing fails after server deployment, with a 404 error reported. Cause: The access permissions for the file upload directory are not configured correctly, or the local temporary file path is not synchronized to the server deployment directory, causing the parsing node to fail to read the uploaded file.
- Symptom: Structured fields are missing from parsed documents. Cause: The structured extraction mode of
customPdfParseis not enabled, and only the default text parsing is used, which cannot extract key fields such as batch number and culture temperature from LIMS spreadsheets. - Symptom: Parsing results are empty or incomplete when pushing files to external systems in bulk. Cause: No associated batch information is specified for the files, causing chunks to fail to match business context.
How to confirm the configuration is correct
- Upload a typical batch release document PDF, and check whether all key fields such as batch number, culture parameters, and test results are included in the parsing results.
- Adjust the
chunkSizeparameter, then check whether the length of the chunked results meets expectations, ensuring that key process steps are not split in a single chunk. - Test uploading a large archive document under 1000 MB, and confirm that no timeout error is triggered during the parsing process.
- Simulate pushing files from an external system, and check whether the parsing node can normally read the file content and complete chunking.
The values provided are common starting points and should be measured against the reader's own samples.