What Data for This Category Looks Like
Daily documents for this category typically include cotton blending or ingredient lists, process setup sheets, equipment operation records, yarn/fabric inspection records, defect records, and similar items. Data sources include raw fiber inspection reports, ERP systems, real-time data from spinning and looms, evenness testers, fabric inspection machine collected data, and order specification files. Update frequency adjusts synchronously with each batch of raw materials, each shift’s production, and each piece of fabric output. Most documents contain structured fields with units, such as fiber grade and blending ratio on cotton blending lists, spinning twist and loom speed on process setup sheets, yarn strength and fabric defect type on inspection records.
Constraints Imposed on Document Parsing and Chunking
Documents in this category have high update frequency and mostly combine structured tables and fields with units. This requires the parsing stage to support incremental updates and table recognition. A single document may contain associated data from multiple batches and shifts. Chunking must retain contextual connections between fields to avoid breaking complete process information for a single production batch. Additionally, document sources are diverse, including Excel, PDF, and text formats exported by equipment. Parsing rules for different file types must be adapted to ensure structured data is not lost. Defect records are strongly tied to fabric lot numbers and shifts. Chunking logic must match the production unit’s granularity to prevent separation of associated information.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapt to the contextual length of unit-bearing field combinations (such as twist + speed + fabric length) in documents of this category, avoiding breaking the associated information of a single set of process parameters |
chunkSplitter | Split by paragraph and table block | Retain the table structure and contextual connections between fields for ingredient lists and inspection records, preventing table data from being split into scattered pieces |
customPdfParse | Enable structured table recognition mode | Adapt to multi-column table data in process setup sheets and defect records, extract complete cell content and hierarchical relationships |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Meet the parsing time requirement for complete cotton blending lists (including multi-page tables) for a single batch of raw materials, avoiding timeout during large file parsing |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Support uploading batch files of multi-equipment operation records for a single batch, meeting business requirements for bulk import |
chunkSettingMode | Manual configuration | Customize chunking rules based on the structured document characteristics of this category, do not use generic default configurations |
Three Common Misconfigurations
- Symptom: After uploading a cotton blending list in Excel format, the knowledge base displays the table but search results do not include table content. Cause: Custom table parsing mode is not enabled, and Excel data is split only by plain text rules, resulting in loss of table structure information.
- Symptom: In chunked documents, defect records for a single piece of fabric are separated from the corresponding fabric lot number and shift information. Cause: Chunk length is set too small, breaking the complete contextual connection of the inspection record for that single piece of fabric.
- Symptom: After processing documents with custom chunking logic locally, uploads to the server have abnormal retrieval matching rates. Cause: Chunking parameters are not unified, and configurations for
chunkSizeandchunkSplitterbetween the server and local environments are inconsistent, leading to vector space mismatch.
How to Verify Configuration Correctness
Based on the configuration logic of FastGPT v4.17.0, configuration effect can be verified through the following actions:
- Upload a single process setup sheet in PDF format, confirm whether the parsing result retains complete paragraphs and associated tables of parameters such as equipment speed and spinning twist.
- Upload a single cotton blending list in Excel format, confirm that the parsed content includes complete fields such as fiber blending ratio and batch information.
- Upload multi-page equipment operation records, confirm that each chunked segment contains contextual information such as shift and equipment number, with no separation of associated information.
- Adjust the
chunkSizeparameter, confirm that the length of chunked segments meets the preset range and does not break the associated content of a single set of process parameters.
The values provided are common starting points, and should be assessed against the relevant sample sets.