What data for this category looks like
Daily documents for this category include coating process cards, coating batch sheets, paint mixing records, film thickness and color difference reports, oven curves, defect repair orders, VOCs ledgers, and more. Data sources include coating supplier TDS/SDS, data collected from spray booth and oven equipment, colorimeters, film thickness gauges, environmental monitoring systems, and MES systems. Data updates occur per coating batch, per shift, or via continuous collection. Some documents contain multi-unit fields: film thickness is measured in μm, coating viscosity in Ford Cup #4 seconds, and VOCs concentration in mg/m³. Some documents use time-series format, such as oven curves, while others use table format, such as process cards.
What constraints do these characteristics impose on the document parsing and chunking process
Multi-source, heterogeneous document formats and multi-unit fields require the parsing process to retain original fields and units to avoid information loss. High-frequency updated real-time data such as oven curves requires chunking to retain time-series context, preventing disruption to process association logic. Differences across document types require chunking configurations to be flexible, adapting to multiple formats including tables, time-series data, and short text. Compliance documents such as VOCs ledgers require precise chunking to ensure no content is missed during subsequent summary work. Matching business requirements for pre-treatment, spraying, and curing windows requires that chunking does not split continuous text related to processes. Doing so will affect the accuracy of subsequent ledger summary.
How to configure the settings
These configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Spraying-related documents include both long paragraphs of process descriptions and short entry-style paint mixing records. This range retains process-related context while avoiding overly long chunks that reduce retrieval efficiency |
chunkSplitter | Custom delimiters + semantic chunking | Documents contain multiple formats including tables, time-series curves, and short records. Using line breaks and semicolons as basic delimiters combined with semantic chunking prevents damage to process logic |
customPdfParse | Enable structured table parsing | Coating process cards and film thickness and color difference reports contain large amounts of structured table data. Enabling this setting preserves field correspondence and improves subsequent summary accuracy |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Multiple documents related to a single coating batch, when combined, may reach a large volume. This configuration adapts to upload requirements for conventional large documents |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large combined documents requires sufficient time. This setting prevents parsing failures due to timeout |
chunkSettingMode | Manual configuration | Spraying-related documents have large format differences. Manual configuration enables precise adaptation to chunking requirements for different document types |
Three common configuration mistakes
- Symptom: Parsing timeout error occurs when uploading PDF documents larger than 300 MB. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter was not adjusted to a value suitable for large documents. - Symptom: After setting custom delimiters, chunks either merge multiple process paragraphs or split individual paint mixing records. Cause: The chunking logic was not adjusted in combination with the
chunkSizeparameter, relying only on fixed delimiters without enabling semantic chunking. - Symptom: When uploading Excel-format coating batch sheets, the parsed result has no valid fields. Cause:
customPdfParsewas not configured to adapt parsing rules for non-PDF documents, and the default parsing failed to recognize the specific field structure of coating-related Excel files.
How to confirm the configuration is correct
- Upload a single coating process card PDF, view the parsed chunk list, verify that the chunk logic matches the preset
chunkSizeandchunkSplitterconfigurations, and confirm that continuous text related to processes is not abnormally split or merged. - Upload a single VOCs ledger Excel file, view the parsed field list, and confirm that core data such as monitoring time and concentration values are correctly extracted.
- Upload an oven curve PDF under 500 MB, confirm that no parsing timeout prompt is triggered, and verify that the
PARSE_FILE_TIMEOUT_SECONDSconfiguration takes effect. - Adjust
chunkSettingModeto automatic mode, upload a mixed-format document again, compare the differences with the manually configured chunking results, and confirm that the configuration logic meets business requirements.
The values provided here are common starting points, and should be evaluated against the reader's own samples.