This document applies to the document parsing and chunking feature of FastGPT v4.17.0.
What data for this category looks like
The core data of vulnerability scanning services comes from scan plans, vulnerability lists, asset lists, CVE databases, and patch announcements produced by vulnerability scanners. Documents processed and generated daily include standardized scan plan templates, vulnerability detail lists with CVE IDs, remediation tickets, and rescan reports. The data update frequency is weekly or monthly. Document fields include asset IP, vulnerability severity, repair status, patch links, and more. The most time-consuming steps in the processing workflow are confirming asset scope, eliminating false positives, and verifying rescan results. Configuration deviations may lead to high-risk vulnerability exposure, missing patches, or audit deductions.
What constraints do these characteristics impose on the document parsing and chunking link?
Document types are diverse, and the standardization of fields varies. Scan plans contain asset range text, while vulnerability lists include both structured fields and unstructured descriptions. Parsing logic must distinguish between structured and unstructured content. Most data sources are fixed-format files exported by scanning tools. Parsing logic must adapt to different export formats. Incremental data from regular updates must support batch parsing to avoid reprocessing old data. Fields include professional terms such as CVE IDs and vulnerability severity. Chunking must retain semantic integrity, and avoid splitting that breaks associated information. False positive elimination and rescan verification rely on document context. Chunking must retain associations between adjacent tickets and reports, to ensure the model can read contextual information.
How to configure the settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Vulnerability scanning documents contain long-form vulnerability descriptions and structured fields. This range retains semantic integrity of vulnerability details and remediation steps, and avoids breaking the association between CVE IDs and repair status |
chunkSplitter | Split by paragraph + heading | Adapts to the heading hierarchy and paragraph structure of scan plans and rescan reports, ensuring each chunk after splitting contains a complete business unit, such as full information for a single vulnerability |
customPdfParse | Enabled | Most vulnerability scanning documents are PDF formats exported by scanning tools. Enabling this configuration optimizes recognition of tables and images, and improves extraction accuracy of structured fields such as asset IPs and patch links |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports batch import of monthly rescan reports and large asset lists, prevents upload failures due to oversized files, and adapts to weekly or monthly batch data processing requirements |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large monthly rescan reports takes a long time. This duration covers the parsing process for most documents, and avoids parsing failures caused by timeout interruptions |
chunkSettingMode | Manual configuration | Adapts to different types of scanning documents, allows adjusting chunking parameters separately for vulnerability lists and remediation tickets, to meet scenario-specific parsing needs |
Three Common Misconfigurations
- No parsing data after configuring a custom parsing link: This occurs because the export format of vulnerability scanning documents is not included in the custom parsing rules, or the parsing service does not have permission to access the link.
- The model cannot read uploaded files after the document parsing node is configured: This occurs because the
UPLOAD_FILE_MAX_SIZEconfiguration value is smaller than the actual uploaded file size, or the file format is not recognized by the supported parsing engine. - Poor table and image recognition for imported Word documents: This occurs because the
customPdfParseconfiguration is not enabled, or chunking parameters are not adjusted for scanning documents with multiple tables and images, resulting in loss of structured content during parsing.
How to Confirm the Configuration is Correct
- Upload a standard vulnerability list document, and check if the parsed text chunks retain complete CVE IDs, vulnerability severity, and repair status fields.
- Test uploading a monthly rescan report larger than 500 MB, and confirm that the upload and parsing process does not trigger timeout or failure prompts.
- Adjust the
chunkSizeparameter to different ranges, compare the number of parsed chunks, and confirm that the chunking logic behaves as expected. - View the parsing logs, confirm that the custom parsing rules are in effect, and there are no error messages about unsupported formats.
The values provided are common starting points, and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.