What the data for this category looks like
Daily documents for this category include sales orders, master production plans, material requirement plans, weekly and daily scheduling sheets, work orders, material shortage lists, and delivery change notices. Data sources cover ERP orders and inventory, BOMs and process routes, equipment capacity, staff shifts, work-in-progress, and supplier arrivals. Data update frequencies include daily, weekly, when an order is created, or when an exception occurs. Most document structures are a mix of tables and text with fixed field combinations. Fields include order numbers, material codes, capacity values, shift time periods, and delivery dates. Corresponding units include pieces, hours, days, batches, and more.
What constraints do these characteristics impose on the document parsing and chunking process
Daily documents contain complex multi-row tables and fixed business fields. Chunking must not split associations between fields, otherwise the business integrity of scheduling sheets and work orders will be damaged. Data is updated frequently and includes exception-triggered updates. The parsing process must support batch fast processing to avoid parsing failures due to timeouts. Document sources are scattered and have diverse formats. Parsing must support multiple formats including PDF, Word, and Excel, while retaining source identifiers required for document traceability. Some documents contain non-standard encoded characters. Parsing must adapt to special encoding requirements to avoid parsing errors.
How to set the configurations
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Documents in this category include combinations of business fields. A value that is too long will lose contextual associations, while a value that is too short will compromise the business unit integrity of work orders and scheduling sheets |
chunkSplitter | Split by paragraph + table recognition | Daily documents contain mixed content of multi-row tables and text paragraphs. This setting retains the association between table rows and avoids splitting business data within table cells |
customPdfParse | Enabled | Some master production plan documents are formatted PDFs. Enabling this setting retains field layout and table structure, improving chunking accuracy |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | When uploading documents such as scheduling sheets and work orders in batches, large file parsing requires sufficient time to avoid timeout failures |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Some enterprise monthly scheduling summary files have large sizes, so this setting adapts to the file size requirements for batch uploads |
chunkSettingMode | Custom mode | Documents in this category have fixed structures and strong business relevance. Manual adjustment of chunking rules better adapts to the association requirements of business fields |
Three common misconfigurations
- After uploading an XLSX-format weekly and daily scheduling sheet, the chunking result splits the equipment, capacity, and shift content in the same row. This occurs because the table recognition chunking rule is not configured, and default character-based splitting destroys the business association of table rows.
- When uploading a delivery change notice in PPT or Word format, an error message "the argument ‘windows-1252’ is invalid encoding" appears. This occurs because the document contains non-standard encoded characters, and custom encoding parsing adaptation is not enabled.
- When uploading more than 10 work order files in batches, the parsing task fails mid-process. This occurs because the value of
PARSE_FILE_TIMEOUT_SECONDSis too low, failing to adapt to the parsing duration requirements for batch large files.
How to confirm configurations are correctly set
- Upload a typical weekly and daily scheduling sheet in XLSX format, check if the parsed chunking results retain the complete association between equipment, capacity, and shift for each row, with no split table cell content.
- Upload a delivery change notice in Word format that contains special characters, verify that no encoding error prompts appear during parsing, and the parsed text fully retains the business content of the original document.
- Upload a single 100 MB master production plan PDF file, confirm that the parsing completion time does not exceed the preset
PARSE_FILE_TIMEOUT_SECONDSduration, and the task status shows success. - Enter the document traceability page of the knowledge base, check any chunked content, confirm that all are labeled with the source document's file name and upload time, with complete traceability information.
The values provided are common starting points and should be measured against the reader's own samples.