What the data for this category looks like
Mine mining and beneficiation ledger data comes from downhole operation site sensors, manual inspection submissions, and exports from beneficiation process systems. Update cycles cover each shift, daily, and weekly. A single ledger file contains multiple sets of operation and equipment operation records for one or multiple mines. Document structures are primarily tables with fixed headers, supplemented by process description paragraphs. Some mines use custom non-standard fields. Fields include equipment number, operation duration, processing volume, safety inspection items, and more. Units include hours, tons, cubic meters, and others.
Constraints on document parsing and chunking
Data from multiple sources has inconsistent formats, including Excel, PDF, scanned documents, and other types. The parsing link must support multi-format adaptation. Frequently updated ledgers create bulk import requirements. Parsing processes must avoid single-file parsing timeouts. Structures with both fixed headers and non-standard fields require the chunking process to retain field association relationships, preventing splitting from damaging data integrity. Multi-level nested mine ledger directories require the parsing link to support recursive traversal of subdirectory content. A large volume of table documents requires the parsing link to optimize table structure recognition capabilities.
How to configure settings
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Single mine beneficiation ledger record length is moderate. Excessively long chunks lose contextual association, while excessively short chunks increase retrieval overhead |
chunkSplitter | Split by paragraph + table | Ledgers contain many process tables and paragraph descriptions. Using the default text splitter damages table structure and field correspondence |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports large monthly summary PDFs or multi-file package uploads, adapting to bulk ledger import requirements |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Large ledger files take longer to parse, avoiding parsing failure due to timeout |
customPdfParse | Enable table enhancement mode | Mine ledger tables mostly use merged cell formats, and the enhancement mode improves table parsing accuracy |
chunkSettingMode | Manual configuration | Adapts to non-standard ledger formats of different mines, allowing flexible adjustment of chunking parameters |
Three common mistakes
- A 413 error appears during file upload, caused by failing to adjust the
UPLOAD_FILE_MAX_SIZEparameter, which exceeds the platform's default limit. - When bulk uploading a multi-level nested mine ledger folder, only the top-level directory content is parsed. This occurs because directory recursive parsing configuration is not enabled.
- Embedded doc attachments in uploaded beneficiation process PDFs cannot be parsed normally. This occurs because the enhanced attachment parsing mode of
customPdfParseis not enabled.
How to verify proper configuration
- Upload a single large ledger PDF that complies with the
UPLOAD_FILE_MAX_SIZElimit, and check whether the parsing status completes normally. - Upload a beneficiation process table document with merged cells, and verify that the parsed text retains the table structure and field correspondence.
- Bulk upload mine ledger files stored in multi-level directories, and confirm that all subdirectory files are included in the parsing queue.
- Call the API to upload ledger files, and check whether the returned results include parsed chunked text content.
The values listed here are common starting points, and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.