What the data for this category looks like
This category of data comes from daily collaboration and outputs from project security response teams, maintainers, and security researchers. It includes four core document types: security advisories, vulnerability reports, fix patches, and CVE application materials. Data sources include private vulnerability reports, code repositories, official channels of CVE numbering authorities, and public release platforms. Updates follow a single vulnerability event cycle: one set of associated documents is generated each time a vulnerability is disclosed or fixed. Documents contain fixed fields such as vulnerability ID, impact scope, reproduction steps, fix solutions, and disclosure time. The length of individual documents varies widely, ranging from a few dozen words in advisory summaries to thousands of words in detailed fix descriptions.
What constraints do these characteristics impose on the document parsing and chunking link
The characteristics of this document category impose multiple constraints on the parsing and chunking process. First, the data includes both private and public materials. The parsing process must distinguish permission scopes to prevent leakage of sensitive vulnerability information. Second, individual documents have wide length variations, with short advisories and long fix descriptions coexisting. This requires adapting chunking logic for different lengths. Third, updates follow event cycles: each new vulnerability event requires full reprocessing of all associated documents. This necessitates consistent and reusable parsing workflows. Additionally, some documents contain code snippets. Parsing must retain code formatting and syntax structure to avoid chunking breaking code readability.
How to set the configurations
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents includes both short advisories and long fix descriptions. This range balances context completeness and retrieval accuracy |
chunkSplitter | Split by paragraph + code block | Documents contain both text and code snippets. Splitting by native structure preserves semantic integrity |
customPdfParse | Enabled | Most core documents for this category are PDF-format security advisories and vulnerability reports. Enabling this allows accurate extraction of text and embedded elements |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Large fix patch documents take longer to parse. This duration covers most scenarios |
UPLOAD_FILE_MAX_SIZE | 500 MB | Some vulnerability reports include full code change packages, so large file upload support is required |
chunkSettingMode | Manual configuration | Documents in this category have large structural differences. Manual configuration allows flexible adjustment of chunking rules |
Three common mistakes
- Issue: After parsing a PDF document, only plain text is extracted, and embedded vulnerability screenshots or code blocks are not recognized. Cause: The
customPdfParseconfiguration is not enabled. The default parsing only extracts plain text and cannot handle graphic-and-text mixed security documents. - Issue: Retrieval results show broken context after chunking, with descriptions of vulnerability impact scope split across two chunks. Cause: The configuration does not follow the recommended
chunkSizerange, or the chunking unit is incorrectly set to a non-character type, resulting in forced splitting of key semantics. - Issue: Timeout errors occur when parsing large vulnerability reports, and the task status shows failure. Cause: The value of
PARSE_FILE_TIMEOUT_SECONDSis too low, and insufficient time is reserved for processing documents with large amounts of code changes.
How to confirm the configuration is correct
- Upload a typical security advisory PDF, and check whether all text and embedded elements in the parsing results are fully extracted.
- Review the chunked document fragments to confirm that key fields such as vulnerability ID and impact scope are not split across different chunks.
- Upload a fix patch document containing code snippets, and verify that the code formatting is retained after parsing.
- Simulate batch processing of vulnerability event documents, and check whether the completion status and time consumption of parsing tasks meet expectations.
The values provided here are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.