What the data for this category looks like
Typical users include security R&D teams, security service providers, and project R&D teams. These teams regularly process documents such as code audit reports, vulnerability proof materials, repair recommendation documents, and retest records. Data sources include project source code repositories, third-party dependency lists, static application security testing (SAST) tool output results, and public vulnerability databases. Document update frequency changes with project versions or audit cycles. Each document typically contains both structured and unstructured content, including risk entry IDs, associated code snippets, risk levels, repair steps, and retest results.
What constraints these characteristics impose on document parsing and chunking
Data sources include code snippets and structured risk entries. Parsing processes must retain code formatting and risk entry boundaries. This prevents critical security information from being split across chunks. Audit document formats vary widely across different projects. Some documents include tabular dependency lists and nested risk descriptions. Flexible parsing rules are required to accommodate these variations. Document update frequency is high. Parsing workflows must support quick adaptation to new document structures. Parsing results must not omit full context for high-severity risks. Otherwise, oversights may occur in subsequent security review steps.
Configuration Settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
customPdfParse | Enabled, bound to pdf-marker parsing rules | Code audit reports often contain code snippets and structured tables. Custom PDF parsing preserves formatting and code block structure, preventing content corruption after parsing |
chunkSize | 800–1200 characters | Code audit documents include long code snippets and complete risk descriptions. This range retains full context for a single risk entry, avoiding splitting repair recommendations from their associated risks |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Audit reports for large projects may include multiple attachments and source code snippets. The timeout range covers the full parsing process to prevent mid-run interruptions |
chunkSplitter | Split by code blocks and headings | Document structure centers on risk entries. Splitting by headings and code blocks ensures each chunk corresponds to an independent security item, facilitating subsequent security localization and review |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Audit reports for large projects may include multiple dependency lists and source code attachments. This setting accommodates large file upload requirements |
chunkSettingMode | Custom mode | Audit document formats vary significantly across different projects. Custom mode allows flexible adjustment of chunking rules to adapt to various security document structures |
Three Common Configuration Mistakes
- Symptom: An empty references field is returned after calling the parsing API. Cause: The
customPdfParseconfiguration is not enabled, so associated reference fields in the document cannot be recognized. - Symptom: Custom parsing configuration options are not displayed after uploading a PDF. Cause: The pdf-marker integration is not enabled in the system settings of FastGPT v4.17.0, and the locally deployed pdf-marker has not completed port connectivity with the platform.
- Symptom: Chunk results show content spliced across multiple risks. Cause: The
chunkSizesetting is too small, merging repair suggestions for one risk with descriptions of other risks into the same chunk, leading to confusion of security information.
How to Verify Proper Configuration
- Upload a test audit report that includes code snippets and risk entries. Check if each parsed chunk corresponds to an independent security item.
- Navigate to the parsing configuration page in system settings. Confirm that the
customPdfParsetoggle is enabled and bound to the correct pdf-marker service address. - Review parsing logs to confirm that no timeout errors corresponding to
PARSE_FILE_TIMEOUT_SECONDSare triggered. - Call the parsing API. Check if the returned result includes associated content in the references field.
The values listed here are common starting points. They should be evaluated against the specific samples used in the target deployment.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.