What the data for this category looks like
Policy documents for power generation groups are mainly sourced from red-headed documents released by the group’s internal compliance department, and industry policy documents issued by state-owned asset supervision and administration authorities and energy industry regulatory authorities. Document update rhythm follows policy release timelines, with no fixed cycle. Updates are synchronized when new policies are released or existing policies are revised. Most documents are in standard PDF or Word format, and include fixed fields such as uniform document numbers, issuing units, effective dates, body clauses and attachments. Some documents come with high-resolution scanned attachments. Unit fields mostly use standard official format expressions for numbers, dates and applicable scopes.
What constraints these characteristics impose on the document parsing and chunking step
The fixed field structure of power generation group policy documents requires the parsing process to extract metadata such as document numbers and effective dates simultaneously. This prevents disconnection between chunked information and the original document. The diversity of document formats (standard PDF, scanned files, Word) requires the parsing engine to support multiple formats. Special attention must be paid to OCR recognition for scanned documents. The lack of a fixed update cycle requires the parsing workflow to quickly adapt to new-format policy documents. Large documents and multi-attachment scenarios require the chunking process to handle long text. This avoids semantic fragmentation caused by overly long single chunks, and ensures that attachment content can be fully parsed.
How to set up the configuration
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Single clauses or sections of power generation group policy documents mostly fall within this range. This ensures complete semantics within each chunk, and avoids logical fragmentation caused by excessive splitting |
chunkSplitter | Split by title + paragraph | Policy documents have clear chapter titles and clause numbers. Splitting by this rule preserves the hierarchical structure of the original document, which facilitates subsequent question-and-answer matching |
customPdfParse | Enabled | Automatically filters non-core content such as headers, footers and redundant page numbers from policy documents, and extracts accurate main text and metadata |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Parsing large policy documents with multiple attachments takes a long time. This range prevents parsing tasks from being interrupted due to timeout |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to policy documents with high-resolution scanned attachments, and prevents large files from being blocked during upload |
chunkSettingMode | Manual configuration | Different types of policy documents (such as industry regulatory policies, internal management systems) have different chunking requirements. Manual configuration allows flexible adjustment of parameters |
Three common mistakes
- Symptom: A connection failure error is returned after uploading a policy PDF. The log shows parsing service connection timeout. Cause: When deploying FastGPT locally via Docker, the marker-pdf container is not added to the same network. This prevents FastGPT from accessing the parsing service.
- Symptom: Key metadata such as document numbers and effective dates are missing from parsed document chunks. Cause: The
customPdfParseconfiguration is not enabled. The parsing engine fails to recognize the fixed fields in the document, resulting in failed metadata extraction. - Symptom: Uploading a policy document that exceeds the preset size is directly blocked during the upload step. Cause: The value set for
UPLOAD_FILE_MAX_SIZEis too low, and does not match the upload requirements of large policy documents.
How to confirm the configuration is correct
- Upload a copy of the group’s internal policy red-headed document PDF. Check whether the parsed text retains complete main text clauses and metadata, with no redundant headers or footers.
- Navigate to the knowledge base configuration page in FastGPT. Verify that the values of configuration items such as
chunkSizeandchunkSplittermatch the preset scheme. - Upload a large policy document with multiple attachments. Check that the parsing task completes within the range set by
PARSE_FILE_TIMEOUT_SECONDS, with no abnormal interruptions. - If a custom parsing script is enabled, upload a test document and review the parsing results. Confirm that the script takes effect and completes the expected format adjustments or metadata extraction.
The values provided are common starting points, and should be evaluated against the reader’s own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.