What the data for this category looks like
Product consultation-related data primarily comes from official financial product manuals, internal customer service question-and-answer repositories, and regulatory public disclosure documents. Updates are triggered by new product launches or adjustments to regulatory policies, with no fixed cycle for routine updates. Document structures include structured rate tables, underwriting rule clauses, and standardized question-and-answer entries. Fields include product ID, expected return rate, underwriting age, and minimum investment amount. Units include percentage, year, yuan, and others.
What constraints these characteristics impose on the document parsing and chunking workflow
If table rows from structured rate tables are forcibly split by line breaks, subsequent vector retrieval will lose associated rate and product information. Therefore, structured table parsing capability must be retained. Long-text product clauses contain nested underwriting conditions and return descriptions. Chunking must preserve contextual logic to avoid breaking clause associations. When standardized question-and-answer entries are mixed with long documents, chunking strategies for short question-and-answer content and long manuals must be differentiated. Field-based information such as expected return rate and minimum investment amount must be bound to their associated product information, and cannot be split into chunks independently.
How to set the configurations
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
chunk_size | 800–1200 characters | Product consultation documents mostly consist of long clauses and structured tables. This range balances contextual integrity and retrieval accuracy |
chunk_overlap | 50–100 characters | Long clauses must retain contextual connections after chunking, preventing critical underwriting or return information from being split across different chunks |
enable_table_parse | Enabled | Product documents contain numerous rate tables and underwriting rule tables. Enabling this option preserves structured table information |
custom_separator | Line break + title marker | Product documents mostly separate paragraphs by titles and line breaks. Combining custom separators enables precise splitting of different logical modules |
max_chunk_length | Calibrated via actual testing | Adapts the maximum allowed chunk length for documents of different formats, preventing individual chunks from exceeding system processing limits |
The parameter values provided on this page are general recommendations to serve as a starting point for configuration. Actual values are affected by material format, data volume, and business rules. Specific issues require individual analysis. It is recommended to test on your own samples before finalizing settings.
Three common configuration mistakes
- Setting
custom_separatorto only line breaks results in chunking outcomes that either merge multiple independent paragraphs or split a single long paragraph into multiple chunks. This occurs because title hierarchy markers from product documents are not used as auxiliary separators. Relying solely on line breaks cannot distinguish different logical modules. - Deploying Marker to parse PDFs via Docker returns the error
{"detail":"Error message: Parsing failed"}with a 500 status code. This happens because environment variables for font packages required for PDF parsing are not correctly configured, or mounting permissions are insufficient, preventing rendering of PDF page content. - Structured rate tables in chunking results are split into scattered text lines with no complete table structure. This occurs because the
enable_table_parseconfiguration is not enabled, causing the parsing stage to split table content directly by text lines.
How to confirm configurations are set correctly
- Upload a typical product manual document, view the parsed chunk preview, and confirm that long clauses are not overly split or merged.
- Upload a document containing a rate table, check if the chunking results retain the complete table structure with no scattered line splits.
- Adjust the
chunk_sizeparameter, re-upload the same document, and confirm that changes in chunk length match expectations. - Verify the
custom_separatorconfiguration, confirming that the separation between titles and body text aligns with the typesetting logic of product documents.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.