What this category of data looks like
The core data for this category comes from order and inventory information, BOM lists, process routes, equipment capacity, staff shifts, work-in-progress ledgers, and supplier delivery details from manufacturing enterprise ERP systems. Daily processed documents include sales orders, master production plans, material requirements plans, weekly and daily scheduling tables, work orders, material shortage lists, delivery change notices, and more. Document structures cover two categories: structured tables and plain text descriptions. Fields include order numbers, material codes, required quantities, capacity values, shift time periods, delivery dates, and others. Units involve pieces, hours, days, and similar metrics. Data update frequency is daily, weekly, or triggered in real time alongside orders or abnormal situations.
What constraints these characteristics impose on the document parsing and chunking workflow
The multi-type structured documents, strongly associated business fields, and high-frequency update characteristics of this category impose multiple constraints on document parsing and chunking. First, scheduling tables and material requirements plan documents that contain large numbers of tables must retain the complete association of fields within the tables, to avoid splitting that breaks business logic. Second, real-time high-frequency update data (such as emergency order additions, material shortage changes) requires the parsing process to have high efficiency, and the parsing time per single file must be controlled. Finally, for business documents with combined multiple fields, chunking must retain contextual associations to ensure that subsequent Q&A can accurately match the corresponding relationships between orders and work orders, scheduling and capacity.
How to configure the settings
The following configurations are formulated based on official functional parameters of FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the field density and contextual association length of documents in this category, avoiding splitting of critical order and scheduling associated information |
chunkSplitter | Split by paragraph + heading | Matches the structured hierarchy of documents in this category (such as work order headers, scheduling table line items) and retains the integrity of business logic |
customPdfParse | Enable table recognition mode | Documents in this category contain a large number of structured tables such as material requirement tables and scheduling tables, and fields and values within tables need to be fully retained |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Addresses parsing time for batch ERP-exported documents, avoiding task interruptions caused by single-file parsing timeouts |
chunkSettingMode | Manual configuration | For the special field units of documents in this category (such as capacity hours, delivery days), customize chunking rules to adapt to business needs |
Three common configuration mistakes
- Uploading a Java interface document with the suffix changed to txt results in empty parsing results. The cause is that the default parsing logic only matches processing rules based on file suffixes, and does not enable custom parsing adaptation for structured text in non-standard formats.
- After calling the
pushdata apito upload data in chunk mode, the interface continuously displays the indexing status. The cause is that thePARSE_FILE_TIMEOUT_SECONDSconfiguration value is too small, or thechunkSizevalue exceeds the reasonable chunking range of business documents, resulting in incomplete parsing tasks. - Uploading a PDF of equipment inspection instructions containing images results in unextracted text content in the images. The cause is that the image text extraction function of
customPdfParseis not enabled, and the default parsing only extracts plain text content.
How to confirm the configuration is correct
- Upload a typical material requirements plan document, and check whether the parsed chunks retain complete field combinations such as order numbers, material codes, and required quantities.
- View the parsing task log to confirm that the parsing time does not exceed the configured
PARSE_FILE_TIMEOUT_SECONDSvalue, and there are no timeout error records. - Upload a weekly and daily scheduling table containing tables, and check whether the table content is fully recognized as structured chunks and not split into scattered characters.
- Test uploading a business document with a non-standard suffix, and confirm that the parsing result contains the core business content of the document with no empty results.
The values provided are common starting points and should be measured against the reader's own samples.