What this category's data includes
Data for this category originates from formula and process libraries, raw material batch records, reactor DCS systems, weighing systems, LIMS laboratory management systems, and warehouse management systems. Documents cover batch production instructions, feeding verification sheets, reaction process logs, cleaning records, test reports, deviation and change orders, and are updated per production batch and key steps. Documents contain fields including feeding volume, reaction temperature, duration, batch number, raw material batch identifier, with standardized units such as mass, volume, and temperature. Some process parameters have confidentiality requirements.
Constraints for the document parsing and chunking workflow
Document structures vary significantly across multiple batches and steps. Document formats exported from different systems are inconsistent, requiring adaptation to differentiated chunking logic. The per-batch update requirement means batch association information must be retained during parsing, to avoid mixing content across batches. Some process parameters have confidentiality requirements, so sensitive field filtering rules must be configured during parsing. Fields within a single batch document are closely linked, so context association between steps must be preserved during chunking, to avoid losing logical chains such as feeding and reaction after splitting. Additionally, parsing and adapting non-standard format documents is challenging, requiring targeted adjustments to parsing configurations.
How to configure the workflow
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Single-batch documents in this category have compact content and closely linked steps. This range preserves complete feeding and reaction step logic, avoiding context breaks after splitting |
chunkSplitter | Switch based on document type | Document formats vary widely across sources. For example, CSV files exported from DCS systems and PDFs generated by LIMS have different structures, requiring matching chunking rules |
PARSE_FILE_TIMEOUT_SECONDS | 120–180 seconds | A single batch document may include multiple pages of reaction records. A longer timeout ensures complete parsing and avoids mid-process interruptions |
customPdfParse | Enabled (for scanned PDFs, takes ~30–60 seconds per page) | Some documents in this category are formatted process PDFs or scanned documents. Enabling custom parsing improves recognition accuracy for non-standard PDFs |
chunkSettingMode | Manual configuration | Chunking parameters must be adjusted for different document types. Manual mode allows flexible adaptation to parsing requirements for multi-format documents |
UPLOAD_FILE_MAX_SIZE | 500 MB | Complete production records for a single batch may include multiple pages of data. This upper limit meets upload and parsing requirements for large-volume documents |
Three common configuration mistakes
- Issue: Renaming a Java interface file to a TXT file extension and uploading it results in no valid data in the parsing result. Cause: The actual document format does not match the file extension. The parsing engine cannot recognize the Java interface code structure, so valid fields cannot be extracted.
- Issue: Using chunk mode to call the pushdata API to upload documents results in the system showing an indexing status for an extended period. Cause: The
chunkSizeconfiguration is too large, orPARSE_FILE_TIMEOUT_SECONDSis set too short. This causes parsing to time out without completion, blocking the indexing process. - Issue: Uploading a process PDF containing images or an HTML document generated by javadoc results in no image-related content in the parsing result, or empty fields. Cause: The
customPdfParseconfiguration is not enabled. The default parsing engine ignores non-text content and does not adapt to the structure of HTML interface documents.
How to verify correct configuration
- Upload a typical single-batch document, and check if the parsing result includes core fields such as batch number and feeding volume to confirm field integrity.
- Upload test documents in different formats (such as PDF, CSV, HTML) to verify that parsing results meet expectations for each format.
- View parsing logs to confirm that parsing duration does not exceed the
PARSE_FILE_TIMEOUT_SECONDSsetting, with no timeout errors. - Upload a test document containing sensitive parameters to verify that confidential fields have been filtered, ensuring sensitive information is not extracted.
The values provided are common starting points and should be measured against the reader's own samples.