What data is included in this category?
Data related to refining production business consultation and response mainly comes from refinery safety operation procedures, equipment operation and maintenance manuals, business approval process documents, window consultation records, and technical transformation project files. Data update cycles vary. Safety procedures are updated annually or after major technical transformations. Daily consultation records are generated in real time. Document types include dozens of pages of long procedures, single-page business guides, and structured table files. Fields include business item number, processing time limit, required materials, and responsible department. Units involved include tons, cubic meters, MPa, hours, and other refining-specific units.
What constraints do these characteristics impose on document parsing and chunking?
A high proportion of long documents requires chunking to retain the complete logic of individual items, and avoid splitting cross-chapter business processes. A large number of structured tables requires parsing tools to retain table structures, and avoid field loss caused by text disorder. A large number of professional units and terms requires the parsing process to not alter professional fields and units. Real-time consultation records and regularly updated procedures coexist, which requires parsing configurations to adapt to documents with different update frequencies, while ensuring parsing efficiency.
How to set the configurations
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Refining business documents contain professional terms and structured tables. An overly large size will lose context association. An overly small size will damage the logical integrity of a single business item. This range is calibrated through actual tests to adapt to professional document structures. |
chunkSplitter | Split by title + paragraph | Refining business documents mostly divide business steps by chapter. Splitting by title can retain the complete logic of a single item, and avoid process confusion caused by cross-chapter splicing. |
customPdfParse | Enable OCR mode | Some old refining safety procedures are scanned PDF files. OCR mode can extract scanned text, and avoid no valid content after parsing. |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Large technical transformation documents for refining projects can reach hundreds of MB per single file. This setting adapts to large file upload and parsing requirements. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Long document parsing requires sufficient processing time to avoid parsing interruption caused by timeout. |
chunkSettingMode | Manual configuration | For different types of business documents such as process documents and specification documents, chunking parameters can be flexibly adjusted to adapt to diverse document structures. |
Three common configuration mistakes
- Phenomenon: After deploying Marker, logs display an OCR error, and the interface shows document parsing failure. Cause: Scanned refining documents have insufficient resolution, or the Marker OCR dependency font library does not adapt to refining professional symbols, resulting in failed text extraction.
- Phenomenon: Feishu knowledge base cannot sync PPT and PDF documents. Cause: The
customPdfParseconfiguration is not enabled, or the uploaded file exceeds theUPLOAD_FILE_MAX_SIZElimit, which triggers an interception in the synchronization process. - Phenomenon: After connecting the Claude model, an error occurs in the file parsing link, but normal chat functions work normally. Cause: The
chunkSizesetting is too large, causing the context passed to the model to exceed the model's supported length threshold, triggering a format error.
How to confirm the configuration is correctly set
- Upload a scanned refining safety procedure PDF, check if the parsed text contains complete professional terms and table content.
- Upload a technical transformation document with a single size exceeding 500 MB, confirm that the upload and parsing process does not trigger timeout or interception prompts.
- Adjust the
chunkSizeto different ranges, test whether the chunked content retains the complete logic of a single business item. - View parsing logs, confirm that OCR error-related errors do not appear again, and the file synchronization status shows success.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.