What this category of data looks like
This category of data primarily originates from source code repositories, dependency lists, SAST tool outputs, and public vulnerability databases. Commonly processed documents include code audit reports, vulnerability proof documents, repair recommendation documents, and retest records. Data updates align with project versions or individual audit task progress, with no fixed cycle. Most documents contain structured fields such as code snippets, risk level identifiers, and repair steps. Some PDF documents include scanned code screenshots, and there is no unified standardized field format.
What constraints do these characteristics impose on document parsing and chunking
Code snippets from source code repositories must retain their syntax structure. Do not break the integrity of functions or logical blocks during chunking. Avoid splitting code that is critical to risk correlation. Vulnerability database data is mostly short text but requires context association. Chunk length must match the total length of risk descriptions and corresponding code. Scanned code screenshots in PDF documents rely on high-precision OCR parsing, which has clear requirements for parsing modes. Multi-source data updates have no fixed cycle, so incremental parsing and incremental indexing must be supported. Document types have large structural differences, so multiple chunking rules must be adapted. Avoid merging content across logical boundaries into a single chunk.
How to configure settings
The following configurations apply to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the length of risk descriptions and associated code blocks in code audit reports, avoids splitting critical logic |
customPdfParse | Enable high-precision mode | Adapts to code screenshots in scanned vulnerability proof PDFs, restores complete text content via OCR |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Covers time requirements for large code repositories or batch parsing of multiple documents, prevents mid-task timeout interruptions |
chunkSettingMode | Manual mode | Customizes chunking rules for different document types such as code files and audit reports, adapts to structural differences in multi-source data |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports uploading large code packages or batch audit documents, meets project-level data import needs |
Three Common Misconfigurations
- Symptom: No search results appear after uploading a scanned PDF, and an OCR Error error is displayed. Cause: The
customPdfParsehigh-precision mode is not enabled. Scanned document code screenshots cannot be recognized normally, resulting in empty chunked content. - Symptom: Some large files fail to parse after batch uploading code repository files. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted. File parsing time exceeds the default threshold, and the task is forcibly terminated. - Symptom: Some models do not trigger the file parsing process in simple mode. Cause: The
chunkSettingModeconfiguration is not adjusted to match the context length requirements of the current model. Chunked content exceeds model limits, and the parsing logic is not triggered.
How to Confirm the Configuration is Correct
- Upload a single scanned vulnerability proof PDF, check the OCR recognition progress in the parsing log, and confirm that the
customPdfParseconfiguration is active. - Upload a code compressed package larger than 100 MB, wait for parsing to complete, then check the task status, and confirm that the
PARSE_FILE_TIMEOUT_SECONDSsetting covers parsing time. - Upload a code file and an audit report PDF separately, check the chunk preview results, and confirm that the chunking rules under
chunkSettingModeadapt to different document structures. - Initiate a knowledge base search test, enter a known vulnerability description keyword, and confirm that the returned results include the corresponding document fragment.
The values listed are common starting points and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.