What the data for this category looks like
Daily documents for this category mainly include cotton blending or ingredient lists, process setup sheets, equipment operation records, yarn inspection reports, grey fabric inspection reports, and defect logs. Data sources include raw fiber test results, ERP systems, real-time data from spinning and looms, evenness tester test data, fabric inspection machine output, and order specification files. Update frequency adjusts based on production cycles. New ledger documents are usually created per batch of raw materials, per shift, and per roll of fabric. Most document fields cover professional production parameters like fiber grade, blending ratio, spinning twist, weaving structure, defect type, roll number, and shift. Units include count, weight per unit area, length, and other industry-specific metrics.
What constraints do these characteristics impose on the "document parsing and chunking" step?
The document characteristics of this category create multiple constraints for the parsing and chunking process. First, multi-source heterogeneous ledger data includes text, structured tables, and a small number of embedded non-text elements. The parsing process must support both structured table extraction and OCR adaptation for non-text content. Second, frequently updated small-batch documents require the chunking workflow to support fast batch processing, to avoid overall summary efficiency being impacted by single-document parsing timeouts. Third, professional production fields such as twist, defect type, and roll number have strong business relevance. Chunking must retain contextual connections within the same document, to avoid breaking parameter logic after splitting. Fourth, some documents have cross-page continuous data, such as cotton blending details for an entire batch of raw materials. Accurate document structure identification is needed to prevent incorrect splitting of cross-page data.
How to set up the configuration
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Documents in this category mostly contain professional parameters and short paragraph records. This range balances parameter completeness and retrieval granularity |
chunkSplitter | By paragraph + punctuation | Documents such as cotton blending lists and process sheets mostly divide content by production steps or parameter items. Splitting this way retains complete business logic |
customPdfParse | Enabled | Documents in this category often contain embedded tables and non-text elements. Enabling custom PDF parsing optimizes structured extraction results |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Parsing multi-page equipment operation records or grey fabric inspection ledgers takes a long time. This range covers parsing needs for conventional documents |
UPLOAD_FILE_MAX_SIZE | 500 MB | Summary ledger documents for a single batch of this category may include multiple pages of equipment records. This upper limit meets batch document upload requirements |
chunkSettingMode | Manual configuration | Document structures vary widely across this category. Manual configuration allows adjusting chunking rules for different document types, to meet the need to retain professional parameters |
Three common errors
- Scenario: After uploading a PDF document with embedded defect photos, the parsed result has no OCR text corresponding to the images. The interface shows parsing succeeded, but the corresponding fields are empty. Cause: The
customPdfParseconfiguration is not enabled. The default parsing logic does not trigger OCR recognition for embedded images. - Scenario: The table dataset option cannot be selected during knowledge base configuration, and structured table documents such as cotton blending lists and grey fabric inspection records cannot be imported directly. Cause: The data source configuration mode that supports table parsing is not switched to, or the current configuration does not grant selection permissions for table datasets.
- Scenario: Parsed chunked content has spliced unrelated professional parameters. For example, one chunk contains both cotton blending ratio and subsequent weaving structure parameters. Cause: The
chunkSplitterrule is not configured correctly, causing chunks to not split by business paragraphs and breaking contextual connections of professional parameters.
How to verify the configuration is properly set
- Upload a typical cotton blending list document that contains embedded images and tables, and check whether table fields in the parsed result are complete and whether image OCR text is extracted normally.
- For a single multi-page equipment operation record document, check whether chunked content is split by business paragraphs, with no forced splicing across pages or parameters.
- Verify that the uploaded file size complies with the
UPLOAD_FILE_MAX_SIZEconfiguration, and confirm there are no parsing failures caused by file size exceeding the limit. - Test batch upload of multiple ledger documents of the same type, and confirm that parsing progress and result generation follow the expected processing rhythm.
The values provided are common starting points and should be measured against the reader's own samples.