This configuration scheme applies to FastGPT v4.17.0.
What the data for this category looks like
Typical data for this category comes from ERP, WMS systems, barcode/RFID collection devices, AGV conveyance systems, and weighing equipment. Daily documents include receiving orders, putaway orders, material picking/return slips, inventory count sheets, lot ledgers, expiration date records, and more. Data update frequencies cover three categories: real-time (such as AGV scheduling data), daily (such as material picking/return summaries), and monthly inventory counts.
Documents typically have fixed headers and multiple business records. Core fields include material code, lot number, storage location, expiration date, and work order association information. Units include pieces, kilograms, cubic meters, and other measurement types.
What constraints do these characteristics impose on document parsing and chunking?
The business data for this category has characteristics of high row count per file, tightly linked fields, and widely varying update frequencies, which impose multiple constraints on document parsing and chunking.
- Excel ledgers with over 15,000 rows or PDF ledgers at the 100,000-character level require the parsing process to have sufficient file handling capacity and processing duration.
- Association information between lots, storage locations, and work orders is core business data and must not be split arbitrarily. Doing so will negatively impact subsequent ledger summarization and material traceability.
- Real-time updated warehouse scheduling data requires the parsing process to have high processing efficiency, to avoid delays that affect the timeliness of account-physical reconciliation.
How to set the configuration
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the length of single business records in warehousing documents, to avoid splitting association information between lots and storage locations |
chunkSplitter | Split by header + row groups | Retains the association between headers and corresponding business data for material picking/return slips and inventory count sheets |
customPdfParse | Enable structured parsing | Adapts to fixed-format PDFs such as receiving orders and lot ledgers, improving field extraction accuracy |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports large Excel or PDF documents for monthly inventory summaries |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the full parsing duration of 15,000-row Excel ledgers |
chunkSettingMode | Manual configuration | Adjust chunking rules for the special field structure of warehousing documents |
Three Common Mistakes
- Phenomenon: Using the default
chunkSizeparameter to parse Excel ledgers with over 10,000 rows results in excessively large single chunk data, increasing vector generation time and reducing retrieval accuracy. Cause: The chunking threshold was not adjusted for the length of single business records in warehousing documents, and the default chunking logic does not match the business data structure of this category. - Phenomenon: Uploading a batch ledger PDF with 100,000 Chinese characters triggers a parsing task timeout. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter was not adjusted, and the default timeout duration is insufficient to complete full parsing of large documents. - Phenomenon: When parsing material picking/return slips, material lot and storage location fields are split into different chunks. Cause: The default character-based chunking rule was used, and structured header-based grouping chunking logic was not enabled, resulting in breakage of critical associated information.
How to confirm the configuration is correct
- Upload a single 15,000-row Excel warehousing ledger, confirm that the parsing task completes within the duration set by
PARSE_FILE_TIMEOUT_SECONDSwith no timeout errors. - Randomly select 3 different types of warehousing documents (such as receiving orders, inventory count sheets, lot ledgers) and verify that the association between material lots and storage locations is retained after chunking.
- Adjust the
chunkSizeparameter to the 800–1200 character range, compare the content length after chunking, and confirm that single chunk data falls within the reasonable range of single business records. - Check the parsing logs of FastGPT v4.17.0, confirm that the extraction accuracy of structured fields (such as lot number, expiration date, storage location code) meets business verification requirements.
The values provided are common starting points and should be measured against the reader's own samples.