What the data for this category looks like
This category of data is sourced from target systems, public vulnerability databases, automated scanner results, and manual verification records. Daily processed documents fall into four types: test authorization letters, penetration test reports, vulnerability reproduction records, and rectification suggestions. Each document type has dedicated fields. For example, authorization letters include authorization numbers and authorization scopes, while reports include vulnerability details and risk ratings. Data is updated upon each project delivery or major version iteration, with no fixed cycle.
What constraints these characteristics impose on the document parsing and chunking link
Test authorization letters are core compliance documents, so key fields such as authorization scope and authorizing entity must be fully retained to avoid breaking field integrity during chunking. Vulnerability reproduction records contain coherent step descriptions, so operational context must be preserved during chunking to prevent splitting that makes the verification process unreproducible. Documents from different sources have notable format differences: authorization letters in Word format, test reports in PDF format, and scan results in CSV format, so multi-format parsing logic must be supported. Data updates have no fixed cycle, so chunking configurations must accommodate length differences for single-project documents ranging from a few pages to dozens of pages.
How to set the configurations
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Single segments of penetration test reports and vulnerability reproduction records typically fall within this range, allowing complete vulnerability descriptions or reproduction steps to be retained |
chunkSplitter | Custom delimiter + paragraph line breaks | Documents in this category are mostly segmented by authorization items and vulnerability entries. Using custom delimiters matches the inherent structure of the documents and avoids forced splitting |
customPdfParse | Enabled | Penetration test reports are mostly formatted PDFs. Enabling this setting preserves text hierarchy and field associations, preventing content disorder |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Large penetration test reports include multiple attachments, so larger file uploads must be supported to avoid errors due to file size limits |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large scan result files takes a long time; this duration covers the parsing needs of most project documents |
chunkSettingMode | Manual configuration | Documents in this category have large structural differences. Manual configuration allows flexible adjustment of chunking rules to adapt to different types of business documents |
Three common mistakes
- Parsing errors occur after uploading a 3 MB PDF file. The cause is failure to adjust the
UPLOAD_FILE_MAX_SIZEparameter; the default threshold is insufficient for large penetration test report files. - Paragraph merging or splitting errors still appear in chunking results after configuring custom delimiters. The cause is failure to match the inherent delimiters of the document, such as not setting blank lines between vulnerability entries as a splitting rule, leading to chunking logic that does not match the document structure.
- Some fields appear empty after parsing. The cause is failure to enable
customPdfParse, which prevents structured fields in formatted PDF documents from being correctly extracted.
How to confirm proper configuration
- Upload a standard penetration test report PDF, and check if the parsed text retains key fields such as vulnerability ID and risk level.
- Adjust the
chunkSizeparameter, then upload a vulnerability reproduction record, and verify that the chunking results do not split complete reproduction steps. - Upload a test file that exceeds the default upload threshold, and confirm that the upload and parsing processes run normally.
- Check the parsing logs to confirm that the
customPdfParseparameter is active, with no parsing results with format disorder.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.