What this category of data looks like
Daily documents for this category include sales orders, master production schedules, material requirement plans, weekly scheduling sheets, work orders, material shortage lists, and delivery change notices. Data is sourced from ERP orders and inventory, BOMs and process routes, equipment capacity, staff shifts, work in progress, and supplier deliveries. Update frequency is daily, weekly, or triggered by orders or exception events. Most documents are structured tables and short text fields, containing order numbers, material codes, capacity values, delivery days, and other content. Units include pieces, hours, sets, shifts, and others.
What constraints do these characteristics impose on document parsing and chunking
The data characteristics of this category create multiple constraints for document parsing and chunking. First, multi-source, heterogeneous document formats (Excel scheduling sheets, PDF work orders, text change notices) require chunking rules tailored to different structures to avoid breaking the integrity of individual business records. Second, frequently updated business data needs an efficient parsing process to prevent repeated parsing and timeouts caused by improper chunking configurations. Third, field associations across processes and resources require chunking to preserve contextual links, so core information for the same work order or scheduling task must not be split.
How to set the configuration
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Most documents in this category are structured tables and short text fields. Storing 1 process schedule or order detail per chunk makes associated retrieval easier |
chunkSplitter | Split by row (for Excel) | Matches the attributes of individual business document records, preventing automatic chunking from disrupting the row-column logic of work orders and scheduling sheets |
customPdfParse | Enabled | Adapts to standardized process route PDFs and exception reports in this category, retaining field layout and hierarchical relationships |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Large scheduling summary sheets and multi-attachment parsing require sufficient time to avoid interrupting the parsing process due to timeout |
chunkSettingMode | Manual mode | Allows separate adjustment of chunking rules for different document types such as Excel order sheets and PDF change notices, adapting to business scenarios |
Three common mistakes
- Phenomenon: Redundant content spanning rows and columns appears after chunking multi-column Excel documents. Cause: The row-wise chunking rule is not specified, and default paragraph-based chunking breaks the individual record properties of structured tables.
- Phenomenon: Parsing tasks automatically load previously uploaded document content. Cause: The mode to only parse the currently uploaded file is not enabled, and the global parsing cache is reused by default.
- Phenomenon: Image attachments cannot be recognized by the model. Cause: Image OCR parsing configuration is not enabled. The current version only supports direct parsing of text documents.
How to confirm the configuration is correct
- Upload a test weekly scheduling sheet Excel file, check that each chunk corresponds to one process scheduling record per row, with no spliced content spanning rows and columns.
- Upload a single work order PDF file, check that the parsing result only contains the content of that work order, with no residual information from other historical documents.
- Upload an exception report document with images, check that the parsing result includes the text content in the images (if OCR is enabled).
- Upload a scheduling summary file larger than 10MB, check that the parsing task does not trigger a timeout error and completes parsing normally.
The values provided are common starting points and should be measured against the reader's own samples.