What Data for This Category Looks Like
Daily documents for this category include SBOM lists, dependency risk reports, license lists, and repair tickets. Data sources include package managers, code repositories, vulnerability databases, and license databases. Update frequencies are daily or per version. Most documents use structured or semi-structured formats, with core fields including component name, version number, license type, and vulnerability ID. Some documents are HTML-formatted interface instructions or JSON-formatted bulk dependency data. There is no unified fixed single-file format, and incremental or full updates are generated with each version iteration.
What Constraints Do These Characteristics Impose on Document Parsing and Chunking
Daily documents have diverse formats, so support for multiple file types such as JSON, HTML, and TXT is required. Some imported files have non-standard formats, such as code files with modified extensions. There are many structured fields with close associations. Chunking must retain association information between components, licenses, and vulnerabilities to avoid logical breaks after splitting. Update frequencies are high, and a single parse may involve multiple bulk files. Parsing efficiency and stability must be ensured. Parsing failures may result in high-risk dependencies not being identified or license violations not being blocked. Failure retry and exception alert capabilities are required.
Configuration Settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the field density of documents such as SBOM lists and license lists, avoiding losing complete associations between components and related information after splitting |
chunkSplitter | Split by semantic blocks | Adapts to semantically associated content such as vulnerability associations and license bindings in dependency reports, avoiding hard character-by-character splitting that disrupts business logic |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses parsing time for large SBOM files or multi-version dependency lists imported in bulk, avoiding timeout interruptions of parsing tasks |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to full dependency lists or multi-batch vulnerability reports imported in a single operation, meeting bulk governance file import requirements |
customPdfParse | Enabled | For HTML-to-PDF documents generated by javadoc or native HTML interface documents, enable custom parsing rules to extract interface fields and code blocks |
chunkSettingMode | Manual configuration | Customize chunking rules for different types of governance documents such as SBOM and license lists, adapting to parsing requirements for structured data |
Three Common Mistakes
- Phenomenon: Importing a Java interface file with a modified TXT extension results in empty parsing results. Cause: Custom parsing rules are not enabled. The default parsing only recognizes general text structures and cannot extract structured fields defined in the interface.
- Phenomenon: Importing a Java interface file with a modified TXT extension results in empty parsing results.
- Phenomenon: After using chunk mode to call the pushdata API for upload, the index status is displayed indefinitely. Cause: The
PARSE_FILE_TIMEOUT_SECONDSconfiguration value is too low, or thechunkSizesetting is too small resulting in too many chunks exceeding the system's parallel processing limit. - Phenomenon: After uploading a PDF-formatted javadoc document, images or interface code blocks are not parsed. Cause: The
customPdfParseconfiguration is not enabled. The default PDF parsing ignores non-text structured content extracted from embedded HTML interface documents.
How to Verify Proper Configuration
- Upload a single standard SBOM JSON file, check whether the parsing result includes core fields such as component name, version number, license type, and vulnerability ID, and confirm that association information is not lost during chunking.
- Call the pushdata API to upload a test file, check whether the task status completes within a reasonable time frame, and confirm that the timeout configuration meets expectations.
- Upload a Java interface file with a modified extension or an HTML-formatted javadoc document, check whether the parsing result includes structured content such as interface methods, parameters, and code blocks, and confirm that custom parsing rules are active.
- Bulk import multiple dependency list files, confirm that the upload size does not exceed the
UPLOAD_FILE_MAX_SIZElimit, and confirm that no files are automatically blocked. This verification process is based on FastGPT v4.17.0.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.