What This Category’s Data Looks Like
Data sources for this category include wood pulp and waste paper batch ledgers, DCS/QCS production control systems, paper machine online measurement equipment, laboratory testing systems, and energy and water treatment systems. Daily generated documents include pulp preparation sheets, beating records, paper machine operation logs, online quality curves, laboratory reports, and sheet break and product change records. Data update frequencies cover three dimensions: real-time, per shift, and per roll. Fields include pulp batch number, beating consistency, paper machine speed, grammage, moisture, thickness, and more. Corresponding units typically include g/㎡, %, mm, m/min, and others.
Constraints on Vector Models and Indexes
This category’s data mixes structured ledgers, semi-structured operation logs, and unstructured curves and reports. Update frequencies vary widely. Indexes must support both batch full updates and near-real-time incremental synchronization. High-speed continuous web production generates cross-horizontal and vertical multi-point data. Indexes must associate time-series and spatial dimension field information. Parsing results from multiple document types need unified chunking rules. This avoids vector embedding bias caused by differences in field units such as g/㎡ and %. Quality traceability requires fast access to historical data. Retrieval latency must stay within acceptable limits to avoid impacting production troubleshooting efficiency.
Configuration Setup
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the average length of texts such as pulp preparation sheets and beating records, avoids overly fragmented chunks that lose context or overly long chunks that cause embedding bias |
| Overlap Length | 100–150 characters | Covers cross-chunk production process correlation information, such as continuous parameter changes during the beating process |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Similarity Threshold | 0.75–0.85 | Filters low-correlation historical records, matches the time-series consistency characteristics of production data |
| Single Search Limit | Top 10 entries | Balances retrieval efficiency and data completeness, covers the main production process records for a single paper roll |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
Three Common Configuration Mistakes
- Phenomenon: Unable to complete vector model configuration via One API. The interface returns an API call failure prompt. Cause: The proxy address and key binding for One API are not correctly configured, causing the model call link to break.
- Phenomenon: The number of returned search results is far lower than expected. Unable to locate sheet break records for a specified batch. Cause: No index filtering rules are set for pulp batch and paper roll number, leading to recall of irrelevant cross-batch data.
- Phenomenon: Vector similarity calculation results after embedding have excessive bias, unable to distinguish sheet data with different grammages. Cause: No normalization processing is performed on numeric fields such as grammage and moisture, causing differences in units and value ranges to affect vector embedding effects.
How to Verify Correct Configuration
- Upload a single pulp preparation sheet document. Check if the length of the chunked text fragments matches the preset chunking rules, with no excessive truncation or redundant splitting.
- Initiate a retrieval request for a specific paper roll batch. Verify that the returned results include associated records for that batch, and the quantity matches the single search limit configuration.
- Call the vector embedding interface to process the parsing results of long-text online quality curves. Confirm that the task completes normally, with no timeout or truncation errors.
- Adjust the similarity threshold configuration. Observe changes in the correlation of search results. Confirm that the threshold setting matches the matching accuracy requirements of the current business.
The values provided are common starting points and should be measured against the reader's own samples.