What this category of data looks like
Typical stakeholders include development teams, Tech Leads, and open source project maintainers. These users regularly process documents including merge requests, code review comments, commit records, and branch strategy guidelines. Data sources cover Git repositories, CI pipelines, static scan reports, and defect trackers. Update frequency is daily or per commit. Document structures vary:
- Merge requests include titles, lists of changed files, review threads, and commit hashes.
- Review comments include comment content, associated code line numbers, and processing statuses.
- Commit records include authors, timestamps, and summaries of change content.
Fields include file paths, line counts, defect IDs, and other metadata tightly bound to the development workflow.
Constraints on the document parsing and chunking stage
The scattered, frequently updated data sources for this category require parsing and chunking workflows to adapt to multiple document formats such as Git and CI logs. This prevents loss of critical metadata due to format incompatibility. Merge requests and review comments contain strongly associated information such as code line numbers and associated defect IDs. Retaining this metadata during chunking is required, otherwise subsequent change impact assessment and review comment traceability will be compromised. Frequently updated commit records and merge requests require parsing workflows to have low latency. Excessive parsing timeout causes data synchronization delays. Document structures vary widely across different sources. Chunking rules must be adjusted on a targeted basis to avoid key information being split or omitted by uniform chunking logic.
Configuration Settings
These configurations apply to FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the text length of merge requests and review comments, avoids splitting context associated with code line numbers |
chunkSplitter | Split by semantic paragraphs | Adapts to the thread structure of review comments and the chronological logic of commit records, preserves the integrity of comment context |
customPdfParse | Enabled | Processes signed PDF review documents and non-standard format export files, preserves document metadata |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing duration of large files such as CI pipeline logs, avoids interrupting parsing tasks due to timeout |
UPLOAD_FILE_MAX_SIZE | 100 MB | Matches the common size of merge request packages and static scan reports exported from Git repositories |
chunkSettingMode | Manual configuration | Adjusts chunking parameters for different source documents such as PRs, CI logs, and defect reports |
Three common configuration mistakes
- Issue: A truncation error with the message "Only support .txt, .m" is returned when calling the Doc2x tool. Cause:
customPdfParseis not enabled. The default parsing engine cannot process signed PDFs or non-standard format documents. - Issue: Content cannot be recognized after uploading a signed PDF review document. Cause: Custom PDF parsing mode is not enabled. The default parsing logic cannot support encrypted signed document metadata.
- Issue: The linked resource cannot be accessed after clicking the reading link returned by the knowledge base. Cause: Associated Git commit hashes or PR numbers were not retained during chunking, causing the linked resource to fail to match the original document.
How to Verify Correct Configuration
- Upload a signed review PDF document, check whether the parsed result retains the document title and signature metadata.
- Upload a merge request export package larger than 10 MB, confirm that the upload size limit error is not triggered for the parsing task.
- Submit a review comment that includes code line numbers, check whether the chunked result includes the associated line number information.
- Batch import merge request data from one week, confirm that all parsing tasks complete within 600 seconds.
The values listed are common starting points, and should be evaluated against the user's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.