What data for this category looks like
Daily documents for this category include LICENSE files, NOTICE files, dependency license lists, and copyright notices. Data sources are code repositories, dependency lists, license texts, and contribution records. Update frequency follows per version or per dependency change. LICENSE files are plain text, containing license type and authorization scope clauses. NOTICE files contain copyright notices and third-party component information. Dependency license lists are structured entries, including dependency name, corresponding license type, and copyright holder. Contribution records are timestamped texts that track copyright ownership changes. Fields include license type strings, copyright holder names, dependency version numbers, and more. There are no unified fixed units, and some fields contain multilingual text.
What constraints do these characteristics impose on the document parsing and chunking link
Data for this category includes multiple formats such as plain text, structured lists, and timestamped records, with a high update frequency. The parsing link must adapt to multi-format inputs. Compliance clauses in LICENSE files are coherent. Chunking must not arbitrarily split cross-clause content, otherwise information will break during compliance verification. Entries in dependency license lists are independent associated items. Chunking must retain the binding relationship between entries and their corresponding licenses to avoid missing associated information during retrieval. Copyright notices must retain complete holder information and authorization scope. Chunking must not lose key compliance fields. Since errors can lead to license violations and blocked commercial use, the parsing link must ensure key information is not lost, avoiding compliance verification failure due to chunking errors.
How to set configurations
The following configurations are based on the parsing logic of FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the clause length of LICENSE files and the scale of dependency list entries, avoiding splitting that breaks compliance clause coherence |
chunkSplitter | Split by semantic blocks | Adapts to the paragraph structure of license texts and the entry association of dependency lists, avoiding splitting across license clauses or dependency entries |
customPdfParse | Enabled | Processes formatted license PDF files, retaining text hierarchy and copyright notice position information |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to upload requirements for large dependency lists and batch license files |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to parsing time for batch license files and large LICENSE files, avoiding task timeout interruptions |
chunkSettingMode | Manual configuration | Adjusts chunking rules for special fields in compliance documents, ensuring key information such as copyright holders and license types is not lost |
Three common mistakes
- Phenomenon: After uploading a docx-format copyright notice document, search results do not include chapter title information. Cause: The
customPdfParseconfiguration is not enabled, causing format parsing to lose title hierarchy. - Phenomenon: After batch uploading dependency license lists, the number of returned search results is less than the number of entries in the source file. Cause: The
chunkSizesetting is too small, splitting a single dependency's license entry into multiple chunks without correct association, causing some entries to be missed during retrieval. - Phenomenon: A timeout error occurs when parsing large LICENSE files. Cause: The
PARSE_FILE_TIMEOUT_SECONDSsetting is lower than the actual parsing time, causing task interruption.
How to confirm the configuration is correct
- Upload a single LICENSE file, check whether the parsed text retains complete license clause paragraphs without forced splitting.
- Upload a license list containing multiple dependencies, verify that the number of entries returned by search matches the number of entries in the source file.
- Upload a large dependency list file, confirm that the parsing task completes within the time set by
PARSE_FILE_TIMEOUT_SECONDSwithout timeout errors. - Check whether the
customPdfParseconfiguration item is enabled, ensuring that chapter title information is retained after parsing formatted copyright notice documents.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.