What the data for this category looks like
This category of data primarily comes from code repositories, CI pipelines, linked Issue links, and contributor-submitted content. Daily documents include PR descriptions, code review comments, CI execution results, and merge records. Data update frequencies are mostly daily or weekly, covering the full process information of a single code change. Most data structures are semi-structured or unstructured text, including fields such as change scope, test status, and conflict markers. There is no unified fixed format, and differences exist based on project maintenance habits.
Constraints on Document Parsing and Chunking
Decentralized data sources and inconsistent formats require the parsing component to adapt to multiple input types such as code repository Markdown, CI logs, and Issue links, and support batch parsing tasks. Daily or weekly update frequencies mean parsing tasks may trigger in batches, requiring reasonable configuration of timeouts and file size limits. Semi-structured PR descriptions are bound to code change content, so chunking must retain contextual associations to avoid splitting that destroys the integrity of the change intent. Error information in CI results must be extracted and chunked separately to facilitate subsequent precise retrieval and positioning of review issues. Linked Issue links must support jump parsing to ensure complete contextual information related to reviews.
Configuration Settings
The following configurations are based on official rules for FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | PR-related documents include code change descriptions and review comments. This range balances contextual integrity and retrieval accuracy, and adapts to semi-structured review data |
chunkSplitter | Paragraph + semantic splitting | PR document change descriptions and review comments belong to different semantic blocks. Splitting by paragraph preserves the integrity of individual review information, and combining with semantic splitting avoids incorrect cross-paragraph splitting |
UPLOAD_FILE_MAX_SIZE | 1000 MB | CI pipeline logs and large PR code change files may occupy large space. This upper limit covers the batch parsing needs of most projects and prevents task failure due to file size exceeding limits |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | When batch parsing multiple PR-related documents, sufficient time must be reserved for format conversion and chunking to avoid interrupting the parsing process due to timeout |
chunkSettingMode | Manual configuration | PR document formats vary greatly across different projects. Manual configuration allows flexible adjustment of chunking rules to adapt to project-specific review data structures |
customPdfParse | Enable adaptive mode | PR-related PDF review documents often contain code blocks and tables. Enabling this configuration improves the accuracy of format recognition and prevents content loss |
Three Common Mistakes
- Symptom: When using a Yuque public share web link as a knowledge base data source, parsing fails and an unsupported data source type prompt is displayed. Cause: The public permissions or format adaptation scope of the link were not confirmed. FastGPT’s parsing rules only support specific types of public web data sources. Yuque links must meet specified sharing and format requirements to be parsed normally.
- Symptom: When parsing PR documents containing embedded images, the images cannot be correctly extracted and sent to the large model. Cause: The image extraction configuration for document parsing was not enabled, or the adaptation rules for image-to-text conversion were not configured, resulting in image content not being included in the chunked data.
- Symptom: When uploading PR-related PDF review documents, the converted Markdown format is garbled and key review comments are lost. Cause: The
customPdfParseadaptive rule was not enabled, resulting in failure to correctly recognize and convert formats such as tables and code blocks in the PDF.
How to Verify Proper Configuration
- Upload a typical PR-related document, check the parsed chunked results, and confirm that the chunk length matches the preset
chunkSizerange, with no obvious semantic fragmentation. - Submit batch parsing tasks for PR documents, monitor task execution status, and confirm that parsing time does not exceed the duration set by
PARSE_FILE_TIMEOUT_SECONDS, with no failed batch tasks due to timeout. - Try uploading Yuque public links and PR documents in PDF format, verify that parsing is successful, and confirm that images and embedded formats are correctly extracted.
- After adjusting chunking configurations, compare chunked results under different configurations, confirm that modifications to
chunkSettingModetake effect and meet the document structure requirements of the project.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.