What this category of data looks like
Data sources for general investment research scenarios include public regulatory documents, third-party industry database export documents, enterprise self-developed interface documents (such as HTML generated by Javadoc), scattered industry whitepaper PDFs, and internal investment research notes. Update frequencies cover irregular (regulatory documents) and periodic (third-party data and self-developed interface documents). Document structures include structured HTML (fixed class and method hierarchies), semi-structured PDFs (with embedded charts and tables), and plain text notes. Fields include class names, parameter types, document numbers, effective dates, and more. Some fields have units such as bytes, ten thousand yuan.
What constraints do these characteristics impose on document parsing and chunking
Structured HTML interface documents require accurate extraction of class and method blocks to avoid incorrect splitting of key logic. Semi-structured PDFs with embedded charts and images require simultaneous extraction of text and OCR content from images. Documents from multiple sources and with multiple structures need to adapt to different parsing rules to prevent generic parsers from losing specific fields such as class names and parameter types. Periodically updated documents need to support incremental parsing to reduce repeated processing costs. Fields with units need to retain their associated relationships to avoid units becoming disconnected from parameters after chunking.
How to set configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
max_chunk_size | 800–1200 characters | Matches the length of method blocks in interface documents and research report paragraphs, avoids splitting individual logical units |
chunk_overlap | 100–150 characters | Retains cross-chunk context, prevents interface parameters and chart descriptions from being split apart |
enable_ocr | Enabled | Adapts to PDF documents with embedded images, extracts text from investment research data within images |
parse_html_strategy | strict | Accurately parses HTML interface documents generated by Javadoc, fully extracts class, method, and parameter information |
parse_timeout_seconds | 600 seconds | Adapts to large-volume batch documents, prevents task failure due to parsing timeouts |
upload_max_size | 100 MB | Supports uploading large industry whitepapers and database export documents |
The parameter values provided on this page are common starting points for configuration. Actual values are affected by material form, data volume, and business rules. Specific issues require targeted analysis. It is recommended to test on your own samples before finalizing settings.
Three common mistakes
- Phenomenon: After uploading HTML interface documents generated by Javadoc, the parsing result only contains a small amount of scattered text or is completely empty. Cause: HTML parsing configuration is not enabled, or a loose parsing mode is used, leading to incorrect judgment of the DOM structure.
- Phenomenon: After uploading a PDF document with embedded images, only page text is extracted, and text content within images is not retrieved. Cause: OCR parsing configuration is not enabled, or the OCR recognition threshold is set too high.
- Phenomenon: Chunking results split a single interface method into multiple fragments, making it impossible to obtain complete parameter descriptions during retrieval. Cause: The chunk length is set too small, and does not match the block structure of interface documents.
How to confirm correct configuration
- Upload a single Javadoc HTML interface document, check whether the parsing result contains complete class names, method lists, and parameter descriptions.
- Upload a PDF document with embedded charts, verify that the parsing result contains both the text around the charts and the image text extracted via OCR.
- Check the background logs of parsing tasks to confirm that no timeout errors or unsupported format error prompts appear.
- Initiate a retrieval test, enter a specific parameter name from the interface document, and confirm that the recalled chunks contain complete contextually associated content.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.