What this type of data looks like
Daily data is used by manufacturing enterprise PMC, planning, shop scheduling, and material control departments. Data sources include ERP orders and inventory, BOM and process routes, equipment capacity, staff shifts, work-in-progress, and supplier deliveries. Updates occur daily, weekly, or when triggered by orders or exceptions. Daily documents include sales orders, master production plans, material requirement plans, weekly scheduling sheets, work orders, material shortage lists, and delivery change notices. These documents contain fields such as order numbers, material codes, capacity units, scheduling priorities, and delivery dates. Some documents are structured tables, while others are unstructured process descriptions.
What constraints do these characteristics impose on the document parsing and chunking process
Multi-source, multi-format data requires the parsing process to support Word, Excel, PDF, and other document types. This prevents loss of structured fields due to format incompatibility. High-frequency update document scenarios require parsing and chunking times to align with business change rhythms. This avoids delaying scheduling adjustments due to timeouts or chunking delays. Structured scheduling sheets and work order documents require chunking logic to retain cross-row table data associations. It must not split the complete field combination of a single business record. Unstructured process specification documents require chunking to retain context for process descriptions and supporting instructions. This prevents information breaks.
How to set configurations
These configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the single-paragraph process description and field length of work orders and scheduling sheets, avoids splitting cross-row table content |
chunkSplitter | Separate by paragraphs and tables | Distinguishes unstructured paragraphs of process specifications from structured tables of scheduling sheets, retains data associations |
customPdfParse | Enabled | Adapts to the PDF format of process specifications, retains process numbers and version information in headers and footers |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Accommodates batches of master production plan and material requirement plan documents |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Meets the time requirements for batch parsing of multiple documents, avoids parsing interruptions for high-frequency updated documents |
chunkSettingMode | Custom rules | Sets chunking boundaries separately for table fields in scheduling sheets, matches business data structures |
Three common errors
- Symptom: A file access link is returned when calling the knowledge base interface. Clicking the link pops up an error "Only support .txt, .m". Cause: The custom parsing switch for the corresponding document format is not enabled. The system grants access permissions for plain text formats only by default.
- Symptom: A process specification document with process diagrams is uploaded. After parsing is complete, the large language model response does not associate image content. Cause: Image embedding parsing parameters are not configured, or the context association between images and adjacent process descriptions is not retained during chunking.
- Symptom: After setting a custom chunk length, the parsed chunks split a single work order record in the scheduling sheet. Cause:
chunkSplitteris not configured to use dual separation by tables and paragraphs, causing the chunking logic to accidentally split cross-row table fields.
How to confirm the configuration is correct
A single work order document is uploaded. The parsed chunks are checked to confirm they fully retain core fields including process numbers, equipment requirements, and working hours, with no cross-row splits. A batch of master production plan documents is uploaded. The upload size is confirmed to not exceed the UPLOAD_FILE_MAX_SIZE limit, and the parsing task does not trigger a timeout. The knowledge base interface is called to obtain file access links. Documents in different formats (Word, Excel, PDF) are verified to be accessible normally, with no format errors. A process specification document with a single image is tested. The large language model response is confirmed to include the process description content corresponding to the image.
The values provided are common starting points and should be validated against relevant samples.