What This Category of Data Looks Like
Data for this category originates from data asset inventory results, database metadata, business system description documents, and public regulatory documents. Four standard document types are produced regularly: data catalogs, classification and grading rule tables, important data identification tables, and review minutes. Updates occur quarterly, or when the corresponding business system changes. Most documents pair structured tables with explanatory paragraphs. Common fields include asset number, sensitivity level, compliance basis, and responsible person. No unified fixed units apply; values adjust based on the specific asset type.
Constraints on Document Parsing and Chunking
Documents from multiple sources have mixed formats, including both structured tables and plain text paragraphs. Parsing engines must support multiple formats to preserve core compliance information. Quarterly updates and changes tied to business systems require parsing configurations to remain stable, avoiding frequent adjustments. Compliance fields in structured tables are mostly fixed items. Chunking must preserve full table integrity, preventing rule entries that span pages from being split apart. Review minutes are typically assembled from short paragraphs. Chunking must avoid splitting related review comments into separate blocks, which would break contextual connections for subsequent compliance reviews. This category covers full data assets excluding personal information. Parsing must accurately identify sensitivity level fields to prevent missing key compliance content.
How to Set Configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the paragraph length of rule entries and review minutes in this category of documents, avoiding splitting core compliance fields |
chunkSplitter | Separate by paragraph + heading | Preserves the hierarchical structure of classification and grading rules, avoiding breaking contextual connections in review minutes |
customPdfParse | Enabled | Adapts to structured tables and mixed-format compliance documents for FastGPT v4.17.0, improving table parsing integrity |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports batch import of quarterly data asset inventory files |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses parsing time for large metadata export files |
chunkSettingMode | Manual configuration | Customizes chunking rules to match compliance document structure for documents with fixed fields in this category |
Three Common Configuration Errors
- Symptom: After uploading a classification and grading rule PDF, the interface shows that the file content cannot be read. Cause: The
customPdfParseconfiguration is not enabled. The default parsing engine cannot adapt to structured compliance tables. - Symptom: After configuring Feishu knowledge base sync, documents in PPT and PDF formats are not retrieved. Cause: Parsing permissions for the corresponding file types are not enabled in the knowledge base sync configuration, or the
UPLOAD_FILE_MAX_SIZEsetting is too small, blocking large files. - Symptom: After deploying the latest version of marker, logs show an OCR error. Cause:
chunkSettingModeis not configured to adapt to OCR recognition for scanned compliance documents, orPARSE_FILE_TIMEOUT_SECONDSis not adjusted to match parsing time.
How to Confirm Configurations Are Properly Set
- Upload a typical classification and grading rule PDF, then check the parsed chunk preview to confirm that table structures are fully preserved and not split.
- Verify the
UPLOAD_FILE_MAX_SIZEconfiguration to ensure its value matches the maximum size of files currently scheduled for import. - After adjusting
chunkSize, confirm that review minute paragraphs are fully retained within a single chunk. - Test uploading a quarterly data asset inventory file to confirm the parsing process does not trigger a timeout limit.
The values listed above are common starting points and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.