What the data for this category looks like
The core data for this category comes from daily work outputs of security service companies, red team engineers, and enterprise security teams, including test authorization letters, penetration test reports, vulnerability reproduction records, and rectification suggestions. Data sources include target system scan results, public vulnerability databases, automated scanner outputs, and manual verification records. Update frequency adjusts along with project progress or major version iterations. The document structure includes fixed business fields, such as authorization scope and subject information in test authorization letters, and vulnerability grading and reproduction steps in penetration test reports. There is no unified fixed format, but clear logical connections exist between business contents.
What constraints these characteristics impose on the "document parsing and chunking" link
Full retention of the authorization scope and subject information in test authorization letters prevents authorization validity ambiguity caused by splitting. Maintaining contextual connections between the steps and associated CVE numbers in vulnerability reproduction records prevents chunking from breaking attack path verification logic. Data formats from multiple sources vary greatly: scanner outputs take the form of structured tables, while manual records are semi-structured text. The parsing process must adapt to field extraction for different formats. Frequently updated project data requires the parsing process to quickly adapt to new documents, while also avoiding vulnerability omissions or risk rating errors caused by improper chunking.
How to set configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Penetration test reports contain long paragraphs of reproduction steps and risk descriptions. Avoid splitting to destroy contextual logic |
chunkSplitter | Split by paragraph + title | Match the structured title hierarchy of test authorization letters and reports, retain the connection between fields such as authorization scope and vulnerability level |
customPdfParse | Enabled | Adapt to non-standard PDF formats exported by scanners, retain original tables, formulas, and manually annotated content |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Large penetration test report PDFs have significant size. Avoid parsing timeouts causing task interruptions |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Support batch import of multiple project documents and scan result files, adapt to team-level data import needs |
Three common mistakes
- Symptom: Complex mathematical formulas fail to return normally, and parsing errors are displayed on the interface. Cause:
customPdfParseis not enabled, and the default PDF parsing engine is used, which cannot recognize LaTeX format formulas in vulnerability reproduction content. - Symptom: Clicking chunk preview on a PDF file in the knowledge base returns unreadable content. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted. Large scan report PDFs are terminated by the system due to parsing timeout. - Symptom: Logs show OCR error after deploying Marker. Cause: The external OCR engine is not configured for
customPdfParsein FastGPT, or thePARSE_FILE_TIMEOUT_SECONDSsetting is insufficient, causing the OCR process to fail to complete.
How to confirm the configuration is correct
Uploading a vulnerability reproduction PDF containing LaTeX formulas allows verification that the chunk preview fully displays the formula content. Uploading a penetration test report PDF with a size exceeding 500 MB allows confirmation that no timeout errors occur in the parsing progress. Synchronizing PPT and PDF files from the Feishu platform end allows confirmation that files are successfully imported into the knowledge base without format loss. Checking the FastGPT v4.17.0 system logs allows confirmation that the customPdfParse module loads normally and has no errors.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.