What does the data for this category look like?
Refining and chemical investment research data sources include internal enterprise DCS system operation logs, production ledger Excel files, industry association public refining capacity reports, and third-party crude oil and chemical product price databases. Data update rhythms vary: Production ledger data updates daily. Static documents such as process manuals only update when processes are iterated. Industry research reports and market data are released irregularly. Document types include multi-page PDF process descriptions, structured table ledgers, and plain-text research report summaries. Fields include unit feed volume, product output volume, and equipment operating duration, with corresponding units of cubic meters per hour, tons, and hours.
What constraints do these characteristics impose on the knowledge base retrieval and recall workflow?
Multi-source heterogeneous data formats require retrieval systems to adapt to parsing rules for different document types, to avoid losing key values in structured ledgers. Differentiated update rhythms require distinguishing full and incremental indexing strategies, to prevent excessive computing resource usage from full refreshes. Differences in document length and structure require chunking strategies that balance the integrity of technical terms and contextual coherence, to avoid splitting that breaks the logic of process descriptions. Multiple field types and units require unified mapping rules, to avoid retrieval result deviations caused by unit mismatches. Text dense with technical terms requires additional vocabulary enhancement, to improve recall accuracy for professional content.
How to set configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunk length | 800–1200 characters | Refining documents are mostly long-text process descriptions. Excessively long chunks will lose contextual connections, while excessively short chunks will damage the integrity of technical terms |
recall count | top 6–10 results | Refining investment research needs to cover multi-dimensional data including processes, markets, and compliance. Too many results will increase context window pressure, while too few will miss key information |
similarity threshold | 0.72–0.85 | Refining technical terms have high matching requirements. Too low a threshold will introduce irrelevant documents, while too high will miss similar process solutions |
UPLOAD_FILE_MAX_SIZE | 500 MB | Refining process manuals are mostly multi-page PDFs, with large average single-file sizes, requiring adaptation for large file uploads |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Large Excel production ledgers contain multiple worksheet data, with long parsing times, requiring extended timeout periods |
incremental update trigger condition | triggered by file modification time | Refining production data updates daily. Triggering by modification time enables accurate synchronization of newly added or modified documents |
The parameter values provided on this page are common recommended starting points for configuration. Actual values are affected by material form, data volume, and business rules. Specific issues require case-by-case analysis, and it is recommended to test on your own samples before finalizing.
Three common pitfalls
- Phenomenon: After uploading a document, the number of displayed chunks changes from the initial value to a higher number, with duplicate fragments appearing. Cause: The
document deduplicationconfiguration is not enabled, and chunking does not perform shard verification based on the document's unique identifier, resulting in the same text block being split and indexed multiple times. - Phenomenon: No recall results when searching for specific refining process parameters. Cause: No professional term synonym table is configured, preventing matching and recall for similar terms such as "atmospheric tower" and "atmospheric distillation tower".
- Phenomenon: After batch creating multiple knowledge bases, the retrieval scope cannot be specified via variable selection. Cause: The optional values of the knowledge base variable are not configured as the unique identifiers of the corresponding knowledge bases, only the knowledge base names are filled in.
How to confirm correct configuration
- Upload a single refining process PDF document, view the parsed chunk list, and verify that the chunk length falls within the preset configuration range.
- Initiate a search for known refining process parameters, and verify that the similarity of recall results falls within the preset interval.
- After configuring an incremental update task, modify a production ledger file, and verify that only the index content of that file is synchronized and updated.
- After configuring multiple knowledge base variables, attempt to switch the variable value, and verify that the retrieval scope switches to the specified knowledge base accordingly.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.