What this category of data looks like
Core data for this category originates from package managers, code repositories, vulnerability databases, and license databases. Update frequency is daily or when new versions are released. Daily processed documents include SBOM lists, dependency risk reports, license lists, and repair tickets. SBOM lists contain structured fields such as component name, version number, and license type. Dependency risk reports mark vulnerability severity and impact scope. License lists clarify compliance requirements for open source components. Repair tickets record specific tasks for dependency upgrades. Data is stored in formats including tables, PDF, and JSON. Most fields focus on third-party component identification, risk severity, and compliance status. This description is based on the functional logic of FastGPT v4.17.0.
What constraints do these characteristics impose on the document parsing and chunking link?
Documents in this category are mostly structured compliance and risk data, with strong correlations between fields. When splitting content, the binding relationship between components and their corresponding risks and compliance requirements must be preserved. The high-frequency update feature requires the parsing process to be efficient, to avoid delaying dependency governance iteration cycles due to parsing delays. Differences in formats across multiple sources require parsing tools to support multiple input types such as Excel, PDF, and JSON. The process must also avoid losing key fields such as vulnerability severity and license type during format conversion. In addition, errors directly impact business compliance and system security. The chunking process must ensure that core risk fields are not truncated or split into unrelated entries.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the field density of SBOM lists and dependency risk reports, avoids splitting complete risk information for a single component |
chunkSplitter | Line break separation + semantic segmentation | Adapts to the dependency entry structure of Excel and the typesetting format of PDF, retains complete context for a single dependency entry |
customPdfParse | Enabled | Extracts structured fields from PDF-format license lists, avoids text chaos caused by native PDF parsing |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to batch-imported historical dependency reports and large SBOM lists, meets the requirements of large-scale asset identification |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Processes large documents containing tens of thousands of dependency entries, avoids task interruptions due to overly long parsing time |
chunkSettingMode | Manual configuration | Fixes chunking parameters, avoids inconsistent chunking results caused by automatic system adjustments |
Three common errors
- When importing Excel-format dependency risk reports, multiple dependency rows are merged into one knowledge base chunk. The symptom is that when viewing the knowledge base entry list, a single entry contains multiple independent component rows from Excel. The cause is that
chunkSplitteris not configured to use line break separation. The default semantic chunking does not recognize Excel's row-level structure. - After importing a dependency report with the same content, the first and second recall results are inconsistent. The symptom is that the same risk description can only be accurately recalled during the first import. The cause is that
chunkSettingModeis not set to manual configuration mode. Automatic system adjustment of chunking parameters causes minor differences in chunk content. - After batch uploading multiple PDF-format license lists, the workflow cannot extract information per individual document. The symptom is that only the overall summary result of all uploaded documents is returned after batch upload. The cause is that
customPdfParseis not enabled, and chunking per document dimension is not configured. The default process merges the content of all uploaded documents.
How to confirm configurations are correctly set
- Upload a single Excel-format dependency risk report, view the knowledge base entry list, and confirm that each entry corresponds to one row of dependency data in Excel. This verifies that the
chunkSplitterconfiguration takes effect. - Upload a large-volume SBOM list, view the parsing task logs, and confirm that no timeout-related error messages appear. This verifies that the
PARSE_FILE_TIMEOUT_SECONDSconfiguration meets document processing requirements. - Import multiple PDF-format license lists, call the information extraction node in the workflow, select filtering by document dimension, and confirm that the content of each list can be extracted separately. This verifies that the
customPdfParseand chunking mode configurations are correct. - Manually adjust the
chunkSizeparameter, re-upload the same test document, compare the number of knowledge base entries under different configurations, and confirm that the configuration item takes effect normally.
The values provided here are common starting points, and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.