What data for this category looks like
Data for this category primarily comes from source code repositories, dependency lists, SAST tools, and public vulnerability databases. Update frequency is adjusted based on project versions or iteration cycles. Daily processed documents include code audit reports, vulnerability proof documents, repair suggestion sheets, and retest records. Documents typically contain fields such as vulnerability ID, risk level, impact scope, and repair steps. No unified mandatory format exists, and some documents include code snippets or screenshots.
What constraints these characteristics impose on the document parsing and chunking link
Data for this category is scattered and has inconsistent formats, with sources covering code repositories, tool outputs, and vulnerability databases. Parsing must adapt to multiple native formats. Update frequency follows project version iterations, so the parsing workflow must support incremental synchronization. Documents include code snippets, risk level labels, and retest records. Chunking must retain logical associations, avoiding splitting vulnerability descriptions and repair steps. Some documents include non-text content, so parsing and chunking logic for images and tables must be supported.
How to set configurations
These configurations are tailored for FastGPT v4.17.0, as detailed below:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the typical length of vulnerability descriptions and repair steps in code security audit documents, avoiding splitting critical logical units |
chunkSplitter | Split by paragraph + code block | Matches the structure of code snippets and text descriptions in documents, retaining the integrity of business logic and vulnerability information |
customPdfParse | Enabled | Supports parsing code audit reports and vulnerability proof documents embedded in PDFs, extracting text and OCR content from embedded images |
UPLOAD_FILE_MAX_SIZE | 500 MB | Accommodates bulk audit report packages exported from large source code repositories, preventing the parsing workflow from blocking oversized files |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Covers the time required for bulk parsing of multiple files, preventing complete parsing workflows from being interrupted by timeouts |
chunkSettingMode | Manual configuration | Allows adjustment of chunking rules for different content types such as code snippets and tables, adapting to the special structure of audit documents |
Three common mistakes
- Issue: Unable to select tabular-format audit datasets in the knowledge base upload interface. Cause: The
customPdfParseconfiguration is not enabled, or the associated configuration for the table parsing module is not completed. - Issue: After uploading an audit report that includes screenshots, the parsing result contains no text content related to images. Cause: Additional configuration for image parsing is not enabled, or the associated settings for
customPdfParseare not completed. - Issue: After bulk importing an old knowledge base, the chunked content on the new server does not match the original server. Cause: Generic document chunking parameters are used, and chunking boundaries are not adjusted for the project dimension of code security audit documents.
How to confirm configurations are correctly set
- Upload a single code audit report, check if the parsed chunks retain the association between vulnerability IDs, risk levels, and corresponding repair steps, with no critical logic split.
- Upload a PDF document that includes embedded images, confirm that the parsing result includes text descriptions or OCR-extracted content corresponding to the images.
- Upload a bulk audit file package under 500 MB, verify that the parsing task completes within 600 seconds with no errors.
- Upload a tabular-format audit document, check if the parsed chunks retain the complete fields and row data of the table.
The values provided here are common starting points, and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.