What Data for This Category Looks Like
Data sources include ERP orders and inventory, BOM lists, process routes, equipment capacity, staff shifts, work-in-progress inventory, and supplier arrival records. Update frequencies include daily, weekly, or triggered by orders or exceptions. Daily documents include sales orders, master production schedules, material requirements plans, weekly and daily production schedules, work orders, material shortage lists, and delivery change notices. Some documents contain tables, formulas, and hierarchical fields. Fields include order numbers, material codes, capacity units, scheduling dates, work order quantities, and more. Some fields have dedicated unit identifiers.
Constraints Imposed on Document Parsing and Chunking by These Characteristics
Mixed multi-type document structure differences require flexible chunking rules to avoid splitting process-related content within tables. Fields with dedicated units require retaining the binding relationship between fields and units during chunking, to prevent loss of business associations during vector embedding. Frequently updated documents need appropriate parsing durations and upload limits to avoid processing efficiency issues from timeouts or insufficient capacity. Cross-document business associations, such as the corresponding relationship between work orders and material shortage lists, require retaining contextual associations during chunking to ensure complete business logic can be matched during retrieval.
Configuration Settings
This configuration is adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the single-page content length of master production schedules and weekly/daily production schedules, avoiding splitting complete associated information of a single process |
chunkSplitter | Paragraph + table delimiter | Adapts to the mixed text and table structure of work orders and production schedules, retaining process and material associations within table rows |
customPdfParse | Enabled | Processes formatted process route and BOM documents, retains field layout and hierarchical structure, preventing format loss |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Accommodates large files such as monthly production schedules and work order summaries imported in batches, adapting to batch processing needs |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing time of material requirements plan files exported from large ERP systems, preventing task termination due to timeout |
chunkSettingMode | Manual configuration | Adjusts chunking parameters for different document types such as sales orders and material shortage lists to adapt to differentiated structures |
Three Common Configuration Mistakes
- Phenomenon: Matching priority of auxiliary data such as material codes and capacity values is lower than main text content during retrieval. Cause: The binding relationship between fields and business context was not retained during chunking, and the weight of core fields and auxiliary data was not distinguished during vector embedding.
- Phenomenon: Parsing times out after uploading large material requirements plan files. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter was not adjusted, and the default timeout duration is insufficient to complete the parsing and chunking process for large files. - Phenomenon: Table formats of production schedules are not rendered correctly in context references, only plain text is displayed. Cause: The
customPdfParseconfiguration was not enabled, resulting in lost document format after parsing, making it impossible to restore the table structure.
How to Confirm the Configuration Is Correct
- Upload a typical master production schedule document, check if the parsed chunks retain complete information of a single process, and do not split key associated fields.
- View the parsing log to confirm that the
UPLOAD_FILE_MAX_SIZEandPARSE_FILE_TIMEOUT_SECONDSconfigurations do not trigger upload restrictions or timeout errors. - Simulate a business query related to work orders, confirm that the search results include complete process, material and capacity information, with no missing fields or format confusion.
- Test different types of documents such as sales orders and material shortage lists to confirm that the
customPdfParseenabled setting retains the original document's layout and hierarchical structure.
The values provided here are common starting points, and should be evaluated against the reader's own samples.