What the data for this category looks like
Documents for internal office and employee services in new energy development mainly come from project feasibility study reports, equipment ledgers, operation and maintenance logs, employee training manuals, and bidding documents. Equipment ledgers contain structured fields such as equipment number, rated power, and operation and maintenance cycle, with units including megawatts, kilowatt-hours, hours, and others. Operation and maintenance logs are updated daily or weekly, feasibility study reports are updated in stages along with project progress, and employee manuals are revised every quarter. Document structures vary significantly, ranging from thousands of words of long-text reports to parameter lists with complex tables. Some old project documents are in scanned format.
What constraints these characteristics impose on the document parsing and chunking link
Long-text feasibility study reports and operation and maintenance logs require avoiding excessive splitting, as this would damage the integrity of key information such as project progress and power generation. For equipment ledgers and bidding documents with complex tables, if structured processing is not performed, table content will be scattered into fragmented text and cannot be used for subsequent retrieval. Old project documents in scanned format require additional OCR support, otherwise valid content cannot be extracted. Frequently updated operation and maintenance logs need to adapt to the parsing efficiency of batch uploads, to avoid parsing failures caused by timeouts.
How to set the configurations
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 1000–1500 characters | Adapt to the information density of new energy documents, avoid splitting key information groups such as equipment parameters and project nodes |
chunkSplitter | Split by paragraph + semantic segmentation | Preserve semantic coherence of long-text reports, while adapting to document structures with tables |
customPdfParse | Enable table structured parsing | Fully extract table content from equipment ledgers and bidding documents, avoid tables being scattered |
UPLOAD_FILE_MAX_SIZE | 500 MB | Adapt to the upload requirements of large project drawings and batch operation and maintenance log packages |
PARSE_FILE_TIMEOUT_SECONDS | 900 seconds | Adapt to the parsing time of large PDF documents, avoid terminating parsing tasks due to timeouts |
chunkSettingMode | Manual configuration | Flexibly adjust chunking rules for different types of new energy documents |
Three common errors
- Missing parsed table content. The symptom is that the parsed result has no structured data corresponding to tables, only scattered row text. The cause is that the table structured parsing function of
customPdfParseis not enabled, causing tables to be split as ordinary text. - Unable to select table datasets. The symptom is that no optional table dataset options appear in the background dataset configuration page. The cause is that the document parsing mode is not switched to the configuration that supports table structuring, so the system does not recognize the table dataset type.
- Incomplete parsed content. The symptom is that after uploading a large operation and maintenance log package, the parsed result only contains the first half of the content, with subsequent content truncated. The cause is that
PARSE_FILE_TIMEOUT_SECONDSis set too short, forcing termination of the document before full parsing is completed.
How to confirm the configurations are set correctly
- Upload a test document containing an equipment ledger table, check whether the table is fully displayed in structured format in the parsed result.
- Upload a single large project report with a size not exceeding 500 MB, confirm that there are no timeout errors in the upload and parsing process.
- After adjusting the
chunkSizeparameter, test whether the chunked text fragments retain complete equipment parameter groups or project node information. - Check the parsing task logs to confirm that there are no error records related to content truncation or timeouts.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.