What this category of data looks like
The data related to document writing and material drafting for refined oil sales mainly comes from gas station inventory Excel reports, fuel card settlement Word documents, PDF invoicing details, oil product purchase and sales contracts, and similar materials. The data update rhythm is mainly daily batches, with some real-time transaction data updated hourly. Document structures are divided into two categories: structured tables and unstructured text. Structured tables include fields such as oil product grade, unit price, sales volume, revenue, settlement time, and units include liters, tons, yuan per liter, and similar units. Unstructured text includes contract clauses, station notices, and similar content.
What constraints do these characteristics impose on the document parsing and chunking link
The massive row data in structured tables requires that chunks should not split single business records excessively, otherwise complete transaction context will be lost. The scenario of uploading mixed document types requires parsing rules to adapt to the structural features of different formats, avoiding splitting Excel by paragraphs and PDF losing the corresponding relationship between table rows and columns. Parsing large files takes a long time, so sufficient processing time must be reserved. At the same time, the chunk length must adapt to the token limit of the vector model to avoid truncation or matching failure when generating vectors. The integrity of core fields such as oil grade and unit price requires that chunks should not split across fields, and the complete field combination of a single record must be retained.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the token limits of mainstream vector models supported by FastGPT v4.17.0, and matches the field length of a single settlement record for refined oil sales, avoiding loss of business context after splitting |
chunkSplitter | Split by row (Excel), Split by paragraph (Word/PDF) | Adapts to the structural features of different document types. Splitting Excel by row can retain the complete fields of a single sales record, avoiding data confusion caused by cross-row splitting |
customPdfParse | Enabled | For PDF invoicing details and reports for refined oil sales, retain table structure parsing, and avoid losing row and column correspondence due to plain text splitting |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the upload requirements of a single batch of 15,000-row Excel or 100,000-character Word documents |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Avoid parsing timeout for large files, and adapts to the parsing time of complex reports |
chunkSettingMode | Manual configuration | For structured data of refined oil sales, custom chunking rules can be used instead of the default general chunking logic |
Three Common Mistakes
- Phenomenon: A single chunk contains more than 20 Excel sales records after chunking, and the vector matching results are scattered and weakly correlated. Cause: The
chunkSizeparameter is not adjusted, the general chunking configuration is used, and the field length of a single record for refined oil sales is not adapted. - Phenomenon: After uploading a 15,000-row Excel inventory report, the system returns an upload failure prompt. Cause:
UPLOAD_FILE_MAX_SIZEis set too small, and it does not adapt to the upload requirements of large-volume structured data. - Phenomenon: After parsing PDF invoicing details, some oil grade and sales volume fields are misplaced. Cause:
customPdfParseis not enabled, and the system splits by plain text by default, losing the row and column correspondence of the table.
How to Verify Correct Configuration
- Upload a single 100-row Excel sales test file, check the chunk preview interface, and confirm that each chunk corresponds to no more than 20 sales records.
- Upload a 100,000-character Word contract document, wait for parsing to complete, and confirm that there is no timeout error message.
- After enabling
customPdfParse, upload PDF invoicing details, check the parsed table structure, and confirm that the row and column correspondence of fields such as oil grade and sales volume is complete. - After adjusting
chunkSettingModeto manual configuration, check the chunk results, and confirm that there are no chunk contents spanning oil grade or settlement time.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.