What this category’s data looks like
Data comes from dye chemical batch records, dye vat control terminals, colorimeters, customer color samples, sewage ledgers, and production work orders. New documents are generated per vat, per batch, and per shift. Daily documents include dyeing formula sheets, chemical weighing records, dye vat timing curves, proofing confirmation sheets, color difference reports, setting process sheets, and sewage ledgers. Document structures are mostly a mix of multi-column parameter tables and text paragraphs. Fields include vat number, batch number, temperature value, feeding amount, color difference ΔE value, discharge concentration, and others. Some fields have dedicated associated units.
What constraints do these characteristics impose on the document parsing and chunking link
Multi-column timing tables require chunking logic to retain field associations within the table, to avoid splitting parameters such as temperature and feeding amount for a single vat into different segments. High-frequency single-vat or single-batch data updates require chunking to define boundaries by production units (vat number, batch number) to prevent mixing cross-batch data. Fields with dedicated units must retain their association during parsing and chunking, otherwise parameter matching will fail in subsequent question answering. Mixed text and table document structures require chunking tools to adapt to both formats. Default plain text chunking logic cannot fully extract structured data from tables. Numeric data in exception reports must be bound to corresponding vat numbers and production times, and chunking must not break these associations.
How to set the configurations
These configurations are based on FastGPT v4.17.0:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Fits the full length of process parameters for a single vat, avoids splitting continuous process data for heating, feeding, and holding temperatures |
chunkSplitter | Split by table + paragraph | Adapts to mixed text and multi-column table document structures, retains field associations within tables |
customPdfParse | Enable structured table parsing | Extracts multi-column tables from dyeing formulas and color difference reports as structured data, retains correspondence between fields and units |
UPLOAD_FILE_MAX_SIZE | 100 MB | Adapts to the size of PDF documents containing long timing dye vat curves, prevents upload failures |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Allocates sufficient time to parse documents containing large volumes of timing data, prevents parsing timeouts |
chunkSettingMode | Manual configuration, calibrate boundaries based on actual testing | For single-batch production documents, customize chunking boundaries to match vat numbers or batch numbers, prevents mixing cross-batch data |
Three common mistakes
- Phenomenon: After uploading multi-column Excel dye vat records, automatic chunking splits each row of data into different segments, and cannot retain complete parameters for a single vat. Cause: Structured table parsing is not enabled, the default chunking logic does not recognize the association between table columns, and only splits by line breaks or character count.
- Phenomenon: When processing an uploaded process sheet, the document parsing task loads previously uploaded documents from the same batch, causing irrelevant data to be mixed into the chunking results. Cause: Global historical document association parsing is not disabled, or the parameter to only parse the currently uploaded file is not set.
- Phenomenon: After uploading a dyeing formula PDF, the parsing result lacks units for dye chemical usage (such as g/L), leading to failure to match accurate parameters in subsequent question answering. Cause: The field retention configuration for custom PDF parsing is not enabled, and numeric fields with units are not specified for extraction.
How to confirm the configuration is correct
- Upload a PDF document of a single-vat dye vat curve. Verify that parsed segments contain complete timing tables and corresponding parameters, with no incorrect cross-row or cross-paragraph splitting.
- Upload multi-column Excel chemical weighing records. Verify that each segment corresponds to a complete row or set of associated data, with no chaotic splitting.
- Upload a single process sheet. Review parsing task logs to confirm that only the currently uploaded file is parsed, with no data from historical documents mixed in.
- Review parsed segments to confirm all fields and dedicated units are retained, such as color difference ΔE values and g/L labels for dye chemical usage.
The values provided here are common starting points, and should be measured against samples relevant to the specific use case.