What the Data for This Category Looks Like
Daily documents for this category include pulp preparation sheets, beating logs, paper machine operation logs, online quality curves, laboratory reports, and sheet break and product change records. Data mainly comes from wood pulp and waste paper batches, DCS/QCS systems, paper machine online measurement equipment, laboratory test results, and energy and water treatment systems.
Update frequencies cover three dimensions: real-time, per shift, and per roll. Fields and units vary across different documents. For example, pulp preparation sheets include pulp ratio and addition amount, with units mostly tons and kilograms. Online quality curves record grammage, moisture, and thickness, with units of grams per square meter, percentage, and millimeters respectively.
What Constraints Do These Characteristics Impose on Vector Models and Indexing
The data characteristics of this category impose multiple constraints on the vector model and indexing process.
Data sources with varied update frequencies require indexes to support incremental updates. Avoid using full reindexing methods to reduce real-time data synchronization delays. Diverse document structures and field units require retaining business field associations during chunking, to avoid breaking business logic such as pulp preparation and quality control.
Time-series documents like online quality curves are relatively short, while long-text documents like laboratory reports have greater length. This requires adaptive chunking parameter ranges. Bidding quotation preparation needs to link historical production data with current batch parameters. Indexes must support cross-document associated recall, and prioritize recalling business data corresponding to abnormal records such as sheet breaks and product changes.
How to Set Configurations
These configurations are adapted for FastGPT v4.17.0:
| Configuration Item | Recommended Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to long-text documents of this category such as laboratory reports, while avoiding over-splitting time-series documents |
| Overlap Length | 100–150 characters | Retains business associations across chunks, such as continuous ratio records on pulp preparation sheets |
| Similarity Threshold | 0.75–0.85 | Distinguishes business-relatedness of pulp properties, paper machine parameters, etc., avoiding recall of irrelevant historical batch data |
| Single Search Limit | Top 8 entries | Balances recall efficiency and information completeness for bidding quotation preparation, covering data from major production links |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Incremental Update Switch | Enabled | Adapts to real-time and per-shift update frequencies, avoiding time costs of full reindexing |
Three Common Misconfigurations
- Symptom: A single file remains in the indexing queue for an extended period, and the indexing progress displayed in the interface stalls. Cause: The file contains ultra-long time-series data such as online quality curves across multiple consecutive rolls, and no adaptive length parameter was set during chunking, leading to a backlog of chunking tasks.
- Symptom: A 60-second timeout error occurs when switching knowledge base indexes. Cause: Incremental update is not enabled, and full reindexing is performed on the dataset with varied update frequencies of this category, exceeding the system's default timeout threshold.
- Symptom: Low-priority knowledge base content is still recalled after setting knowledge base priorities. Cause: The global cross-knowledge base recall switch is not disabled, causing the system to simultaneously search all bound knowledge base resources.
How to Verify Proper Configuration
- Upload a single typical document such as a pulp preparation sheet. Confirm that indexing completes without errors, and the number of chunks falls within the conventional length range for business documents.
- Enter a query statement containing pulp ratio or paper machine operation parameters, and check whether the similarity scores of the recall results fall within the preset threshold range.
- Trigger a knowledge base incremental update, modify a piece of real-time production data, and verify that updated content can be recalled in subsequent queries.
- Test cross-knowledge base retrieval with priority settings, and confirm that only relevant content from the specified priority knowledge base is recalled.
The configuration values provided here are common starting points, and should be evaluated against the reader's own samples.