What Data for This Category Looks Like
This category of data primarily comes from code repositories, dependency lists, license texts, and contribution records. Commonly processed documents include LICENSE files, NOTICE files, dependency license lists, and copyright notice texts. Data updates are triggered with each version release or each dependency change. Most documents use plain text or structured formats. Dependency license lists often list dependencies and their corresponding license agreements in table form. Copyright notice files contain contributor attributions and descriptions of license scope.
What Constraints Do These Characteristics Impose on Document Parsing and Chunking
Since data sources include scattered text from code repositories and structured dependency lists, parsing must distinguish between plain text license terms and structured table content. This prevents chunking from breaking associated information in tables. Dependency updates occur frequently, so chunking must adapt to small-volume data that changes often. An excessively large chunk length is not recommended. Copyright notice and NOTICE files often contain fixed-format attributions and license statements. Chunking must retain contextual associations, and avoid splitting cross-clause license information into different chunks. Additionally, some license texts contain long legal paragraphs. Chunking must balance semantic completeness and retrieval granularity, to ensure that corresponding clauses can be quickly located during compliance checks.
Configuration Settings
These configurations apply to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the semantic block length of license texts and dependency lists, avoids splitting legal clauses or dependency association information |
chunkSplitter | Split by paragraph and heading | Matches the structured format of LICENSE and NOTICE files, retains contextual associations |
customPdfParse | Enabled | Extracts text via OCR when processing scanned license PDFs, adapts to non-plain-text formats of compliance materials |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing time required for large dependency license lists, prevents parsing failures due to timeout |
chunkSettingMode | Manual configuration | Adjusts chunking rules for different types of compliance documents (such as plain text licenses, structured dependency lists) to fit the unique data characteristics of this category |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to upload requirements for large aggregated files of multiple dependency licenses, prevents blocking due to oversized files |
Three Common Configuration Mistakes
- Symptom: Text fields are empty after uploading a scanned license PDF, and compliance clauses cannot be retrieved. Cause: The
customPdfParseconfiguration is not enabled, and the OCR recognition process is not triggered. - Symptom: After uploading a dependency license list, chunking splits the license information of the same dependency, leading to incorrect index matching. Cause: The
chunkSizesetting is too small, splitting the associated dependency name and license agreement into different chunks. - Symptom: After calling the API to upload a compliance document, the chunked content does not include the header copyright notice of the LICENSE file. Cause: The
chunkSplitteris not configured to retain headings and preceding paragraphs, leading to omission of core notices.
How to Verify Correct Configuration
- A test LICENSE file may be uploaded, the parsed chunk list reviewed, and verification completed that each chunk contains a complete semantic paragraph.
- A scanned license PDF may be uploaded, and the parsed result checked for retrievable text content to confirm that
customPdfParseis active. - A structured dependency list file may be uploaded, and verification performed that chunking retains the association between dependencies and their corresponding license agreements.
- Parsing task logs may be reviewed to confirm that the
PARSE_FILE_TIMEOUT_SECONDSsetting does not trigger timeout errors.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.