What this category of data looks like
Market data primarily comes from official public exchange APIs, structured export files from compliant market data aggregation services, and daily post-market batch snapshot documents. Update cadence: Individual stock market data is pushed in real time during trading hours, post-market data is updated once daily, and exchange-traded fund market data is updated in sync with trading hours. Most documents are structured tables containing fields such as ticker code, ticker name, transaction price, total trading volume, and price change. Units are as follows: ticker code has no unified unit, ticker name is a string, transaction price is in yuan, total trading volume is in shares, and price change uses a numerical unit.
What constraints do these characteristics impose on document parsing and chunking?
The structured table format of market data requires the parsing stage to preserve the correspondence between columns and rows, to avoid generic text parsing breaking field associations. The real-time incremental push feature requires chunking to support splitting by individual ticker or single batch ticker granularity, to reduce resource consumption from repeated parsing of full files. Numeric fields with clear units require parsing to bind fields to their units, to avoid confusion between parameters of different tickers during retrieval. The large volume of post-market batch snapshot files requires chunking to support sharding by ticker code, to lower memory usage during single-block parsing.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
PARSE_TABLE_ENABLE | Enabled | Most market data is in structured table format. Preserving table structure improves field relevance during retrieval |
chunk_size | 800–1200 characters | Market data fields for a single ticker occupy approximately 300–500 characters. This range can hold complete information for 2–3 tickers, avoiding overly fragmented or overly long chunks |
PARSE_INCREMENTAL_ENABLE | Enabled | Real-time market data is updated incrementally. Incremental parsing reduces redundant computation and storage usage |
PARSE_FILE_MAX_SIZE | 500 MB | Single batches of post-market snapshot files typically fall within this size range. Exceeding this limit triggers parsing timeouts |
TABLE_FIELD_BIND_UNIT | Enabled | Market data fields have clear units. Binding fields to their units avoids unit confusion between different tickers during retrieval |
PARSE_TIMEOUT_SECONDS | 300 seconds | Large batch files require sufficient time for parsing, to avoid interrupting the parsing process mid-execution |
The parameter values provided on this page are common starting points for configuration setup. Actual values are affected by material format, data volume, and business rules. Specific scenarios require individual analysis. It is recommended to test on your own samples before finalizing settings.
Three Common Misconfigurations
- Phenomenon: After importing an Excel-format market data dataset, retrieval fails to match ticker codes with corresponding market data. Cause: The
PARSE_TABLE_ENABLEconfiguration is not enabled. Generic text parsing splits tables into scattered text, breaking the correspondence between fields. - Phenomenon: Garbled Chinese text appears when parsing scanned market data documents. Cause: No Chinese language pack for OCR recognition is configured, leading to incorrect Chinese text recognition.
- Phenomenon: Chunked market data displays correctly in the knowledge base preview, but only scattered text is returned when generating responses, with no table structure preserved. Cause: The
TABLE_FIELD_BIND_UNITconfiguration is not enabled, and the binding between fields and units is not preserved, leading to loss of structured information during output.
How to Verify Proper Configuration
- Upload a single structured market data file, check the knowledge base preview interface, and confirm that the table structure is complete and fields match the original document.
- Submit a retrieval request for a single ticker, confirm that the returned results include all preset fields for that ticker, and that fields and units are correctly bound.
- Upload an incrementally updated market data file, check the parsing logs, and confirm that only newly added content is included in the knowledge base, with no records of full repeated parsing.
- Upload a scanned market data document, check the parsed text content, and confirm that there is no garbled Chinese text.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.