What this category of data looks like
This category of data comes from dyeing and finishing factories, fabric enterprises' dyeing, setting, laboratory and quality departments. It covers documents such as dyeing formulas, chemical weighing records, dyeing cylinder temperature-time curves, sample confirmation sheets, color difference reports, setting process sheets, and sewage ledgers. Data sources include customer color samples, formula libraries, dyeing chemical batch records, dyeing cylinder control systems, colorimeters, setting machines and environmental monitoring equipment. The update frequency is per cylinder, per batch, per shift. Documents include fields such as bath ratio, dye dosage, temperature rise curve parameters, color difference values, COD indicators. Units include grams, liters, degrees Celsius, hours, and others.
Constraints on document parsing and chunking
The multi-type documents, high update frequency and strongly associated field characteristics of this category create multiple constraints for document parsing and chunking. First, documents include table formats (dyeing cylinder temperature curves, chemical weighing records), numerical report formats (color difference reports) and long-text process sheets. Parsing logic must adapt to multiple formats to avoid losing table structures or numerical associations. Second, high-frequency updated single-cylinder or single-batch data requires chunk granularity that fits the complete logic of single-cylinder processes. Chunks that are too large cause context redundancy, while chunks that are too fragmented break process associations. Finally, the binding of professional fields and units requires retaining the correspondence between parameters and units during parsing, and not splitting key parameter combinations during chunking.
How to configure settings
This configuration is adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents contains the complete logic of single-cylinder processes. This granularity can cover single-segment process parameters and associated instructions, avoiding splitting content across key steps |
chunkSplitter | Split by paragraph + table row | Dyeing and finishing process documents mostly use paragraphs to explain process steps and tables to record batch data. Splitting according to this rule can retain the integrity of single-segment processes and the independence of tables |
customPdfParse | Enabled | Most documents in this category are formatted process sheets and dyeing cylinder curve PDFs. Enabling custom parsing can retain table structures and field correspondence, avoiding format confusion |
UPLOAD_FILE_MAX_SIZE | 500 MB | Single-cylinder dyeing cylinder curve logs may contain multi-period data. Larger files require upload support, and this value can cover the size requirements of most single-batch documents |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Parsing large dyeing cylinder curve PDFs requires a long time. This range can adapt to the parsing duration of most documents, avoiding mid-process timeouts |
chunkSettingMode | Custom mode | Documents in this category have large format differences. Custom mode can flexibly adjust chunking rules to adapt to the parsing needs of different types of documents |
Three common configuration errors
- Phenomenon: A 504 status code is returned when uploading a dyeing cylinder curve PDF due to parsing timeout. Cause: The value of
PARSE_FILE_TIMEOUT_SECONDSdoes not match the parsing duration of large log documents, interrupting the parsing process. - Phenomenon: When calling the API to retrieve chunked content, associated data for the specified process step cannot be located. Cause:
chunkSplitteris not split using paragraph rules, splitting parameters and instructions for a single process segment into different chunks. - Phenomenon: An upload failure prompt appears when uploading a single-batch sewage ledger, with the error indicating the file is too large. Cause: The value of
UPLOAD_FILE_MAX_SIZEis smaller than the actual size of this document type, failing to adapt to the high-frequency updated batch document requirements of this category.
How to verify correct configuration
- Upload a single-cylinder dyeing formula PDF, check the parsed chunk list, and confirm that single-segment process content is not split across multiple chunks.
- Upload a dyeing cylinder curve log larger than 200 MB, confirm that the upload succeeds and parsing does not time out.
- Call the chunk query API to verify that the index and corresponding content of the specified chunk can be retrieved.
- Upload a setting process sheet with tables, confirm that the table structure remains intact after parsing, and fields and units are not lost.
The values provided are common starting points and should be measured against the reader's own samples.