What this category of data looks like
Data for this category primarily originates from Git repositories, CI pipelines, static scanning reports, and defect databases. Daily processed documents include merge requests, code review comments, commit records, and branch strategy descriptions. Data fields include document ID, submitter, list of changed files, review status, commit hash, and changed content, among others. Update frequency is daily or per commit, and varies dynamically with the development workflow. Most document structures are mixed-format, containing natural language descriptions and code snippets, with some documents featuring hierarchical review comments.
What constraints do these characteristics impose on the document parsing and chunking link?
The characteristics of this category of data impose multiple constraints on document parsing and chunking. First, data is updated frequently with commits, so the parsing process must support frequent changes to avoid delaying development workflows due to parsing delays. Second, documents contain a mixed structure of natural language and code snippets, so chunking logic must preserve the integrity of code blocks to avoid splitting that breaks code semantics. Third, data sources include outputs from multiple tools, so parsing must be compatible with different document formats and cannot rely on a single parsing rule. Fourth, change impact judgment relies on contextual association, so chunk length must balance semantic integrity and retrieval efficiency. Excessively long chunks reduce retrieval accuracy, while excessively short chunks fail to cover complete change context.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents includes natural language review comments and code snippets. This range balances semantic integrity and retrieval efficiency while avoiding splitting that breaks code block structures |
chunkSplitter | markdown | Adapts to the fact that most documents in this category use Markdown format, preserving the original structure of headings, code blocks, and lists |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Large merge requests may include dozens of changed files. This duration covers parsing time for most scenarios and prevents task interruptions due to timeouts |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Meets the upload requirements for large associated documents from open-source projects or large teams, avoiding inability to submit complete merge request documents due to file size limits |
customPdfParse | false | Core data for this category comes from Git-native format documents, so custom PDF parsing does not need to be enabled, reducing unnecessary parsing overhead |
Three common mistakes
- Issue: File parsing API calls return 504 timeout errors. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter was not adjusted, and the default configuration duration is insufficient to handle parsing tasks for merge requests with a large number of changed files. - Issue: Parsed chunks lose complete code snippets. Cause:
chunkSplitterwas not set tomarkdown, and the default chunking logic splits code block delimiters, breaking code semantic integrity. - Issue: File parsing functionality fails after upgrading to FastGPT v4.9.0. Cause: The default parsing logic of the new version was not adapted, and old version custom parsing configurations were still used, resulting in failure to correctly identify merge request document structures.
How to confirm configurations are set correctly
- Upload a sample merge request document that includes code snippets and review comments, and check if the parsed chunks retain the complete format and hierarchical structure of code blocks.
- Submit a test merge request document with 10 or more changed files, and verify that the parsing task completes within the duration set by
PARSE_FILE_TIMEOUT_SECONDS. - Call the file parsing API to confirm that the returned chunk data includes complete changed content and review comments, with no empty or missing fields.
- Check the FastGPT runtime version identifier to confirm that the currently deployed version is v4.17.0, ensuring functional compatibility.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.