What the data for this category looks like
The core data for vulnerability scanning services comes from internal vulnerability scanners, enterprise asset inventories, public CVE databases, and patch bulletins. Four types of documents are produced daily: scanning plans, vulnerability lists, rectification work orders, and rescan reports. Scanning plans include task ID, scan scope, and execution configuration fields. Vulnerability lists include CVE IDs, risk levels, and affected asset fields. Rectification work orders include associated vulnerabilities and rectification requirements fields. Rescan reports include rescan time and repair verification results fields. Data update frequency is weekly or monthly, synced with scan task execution and vulnerability database updates.
What constraints do these characteristics impose on the document parsing and chunking stage
Documents in this category come from diverse sources, including structured tables, plain text, PDF reports, and more. Parsing must adapt to the structural characteristics of different documents to avoid splitting key associated fields such as CVE IDs and risk levels. When batch processing weekly or monthly updated scan task documents, control per-document parsing duration and chunk count to prevent timeout interruptions. Data formats vary widely across sources. Public CVE database entries and internal scan reports have different field layouts. Adapted chunking rules must be configured to ensure complete retention of key vulnerability information. Additionally, parsing accuracy for high-risk vulnerability data directly impacts subsequent security operations. Chunking must prioritize retaining high-risk vulnerability entries to avoid loss of critical information due to chunk splitting.
How to configure settings
The following configuration items can be adjusted in the technical document parsing module of FastGPT v4.17.0:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category’s vulnerability descriptions and rectification requirements are mostly long texts. A chunk length of 800–1200 characters preserves the integrity of individual vulnerability information and avoids splitting key associated fields |
chunkSplitter | Split by paragraphs and headings | Documents such as scanning plans and vulnerability lists have clear heading and paragraph hierarchies. Splitting by this rule preserves the logical structure of the document, facilitating subsequent retrieval of associated vulnerability information |
customPdfParse | Enabled | Most rescan reports for this category are in PDF format. Enabling custom PDF parsing correctly extracts structured fields such as CVE IDs and risk levels from tables, avoiding field loss caused by plain text parsing |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Rescan reports for batch scan tasks may contain large amounts of asset scan data. A 1000 MB upload limit meets the upload requirements for batch documents |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Large rescan reports take longer to parse. A 600-second timeout setting prevents task interruptions caused by parsing timeouts |
chunkSettingMode | Manual configuration | This category’s document structures vary widely. Manual configuration allows adjusting chunking rules for different document types, adapting to the parsing needs of documents such as scanning plans and vulnerability lists |
Three common mistakes
- Phenomenon: After configuring Feishu knowledge base synchronization, when uploading PPT or PDF documents to the knowledge base, the chunking preview interface returns "Unable to read the file content". Cause: The
customPdfParseconfiguration is not enabled. The default parsing rule cannot adapt to the encrypted format and table nested structure of Feishu-synced documents. - Phenomenon: When deploying FastGPT v4.8.12 locally, uploading a vulnerability report containing complex mathematical formulas results in chunked content that fails to correctly display the formula structure. The online v4.9.0 version can normally return formula parsing results. Cause: There are functional differences in the built-in document parsing modules across versions. Earlier versions do not adapt to complex formula typesetting recognition and OCR logic.
- Phenomenon: An error occurs when running the command
docker run --gpus all -itd -p 7231:7231 marker_images:v0.1to start the parsing service. Cause: The startup command parameter format is incorrect, or the local GPU driver version does not match the requirements of the image, causing the container to fail to properly start the parsing process.
How to confirm that the configuration is correct
- Upload a single scanning plan document, check if the parsed chunked content retains key fields such as task ID and scan scope, with no obvious content loss.
- Adjust the
chunkSizeconfiguration, upload a document containing long-text vulnerability descriptions, and verify that the chunk length matches the preset value range. - Enable the
customPdfParseconfiguration, upload a rescan report PDF, and check if structured fields such as CVE IDs and risk levels in the table are correctly extracted. - Upload batch documents, confirm that the parsing task does not experience timeout interruptions within the preset
PARSE_FILE_TIMEOUT_SECONDSduration.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.