What the data for this category looks like
Data sources include dyeing and finishing factories, and dyeing, setting, laboratory, and quality departments at fabric companies. Daily documents cover dyeing formulas, chemical weighing logs, dye vat curves, sample confirmation reports, color difference reports, setting process sheets, and sewage ledgers. Data comes from customer color samples, formula libraries, dye chemical batch ledgers, dye vat control equipment, colorimeters, setting machines, and environmental monitoring systems. Updates occur once per vat, per batch, or per shift. Documents include precise numerical fields with clear units, time-series dye vat operation curves, and compliance records. Significant structural differences exist across different document types.
Constraints on document parsing and chunking
High-frequency updates per vat, batch, or shift require document parsing to respond quickly, to avoid slowing down supplier and process quality audit workflows. A mix of multiple document types requires chunking strategies tailored to different document structures. For example, complete time-series segments of dye vat curves must not be split, and key numerical values and units in dyeing formulas must not be truncated. Precise numerical fields and compliance records require parsing to retain full information, to avoid data breaks caused by chunking. Some documents such as dye vat curve PDFs contain large numbers of embedded time-series charts, which place higher requirements on the parsing tool’s table and image recognition capabilities.
How to Set Configuration Parameters
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the text length of documents such as dyeing formulas and color difference reports, avoids splitting key numerical values and unit fields |
chunkSplitter | Split by paragraph + time block | Preserves logical continuity of dye vat curves and setting process sheets, prevents time-series data from being split across chunks |
customPdfParse | Enabled | Identifies embedded formulas, tables, and dye vat curve charts in PDF documents, improves parsing accuracy |
PARSE_FILE_TIMEOUT_SECONDS | 120–180 seconds | Adapts to parsing time for PDF dye vat curves with multiple vat time-series datasets, prevents parsing failures due to timeout |
UPLOAD_FILE_MAX_SIZE | 500 MB | Supports batch uploading of large-volume documents such as dye chemical batch ledgers and sewage ledgers |
chunkSettingMode | Manual configuration (overlap length 50–100 characters | Retains contextual association between chunks, prevents loss of key numerical values in dyeing formulas across chunks |
Three Common Misconfigurations
- When
customPdfParseis not enabled, uploading a dyeing formula PDF with embedded formulas results in a "failed to read file content" notification. This occurs because custom PDF parsing is disabled, so embedded formulas and table structures cannot be recognized. - When the
chunkSizeparameter is not adjusted and chunk length is set too short, complete temperature rise, material addition, and holding time segments of a dye vat curve will be split across two chunks. Subsequent retrieval cannot associate the complete dataset. This occurs because the chunk length does not meet the logical integrity requirements of time-series documents. - When the deployed marker version does not match FastGPT v4.17.0, parsing a PDF of dye chemical batch records will show an ocr error in logs. This occurs because version compatibility issues cause failure in the OCR parsing step.
How to Confirm Configuration is Correct
- Upload a dyeing formula PDF with embedded formulas, check the chunk preview interface, confirm that formulas display normally and no error prompts appear.
- Upload a single-vat dye vat curve PDF, check the chunking results, confirm that complete temperature rise, material addition, and holding time segments are not split across chunks.
- Check the backend parsing logs, confirm that no timeout errors occur for documents under 500 MB, verify that the
PARSE_FILE_TIMEOUT_SECONDSsetting is effective. - Test Feishu Knowledge Base synchronization of PPT and PDF documents, confirm that the synchronization task completes and the chunk preview loads file content normally.
The values provided are common starting points and should be measured against the reader's own samples.