What the data for this category looks like
Pharmaceutical distribution ledger data comes from inventory and sales systems, logistics delivery receipts, and supplier reconciliation documents. Updates occur in real time alongside transactions or via daily batch synchronization. Documents are primarily structured CSV files and semi-structured scanned PDFs. Fields include generic drug names, approval numbers, batch numbers, inbound/outbound quantities, settlement amounts, circulation nodes, and partner information. Units cover various package measurements such as boxes, vials, cases, and pieces. Some text contains special characters such as hyphens and slashes in approval numbers.
What constraints these characteristics impose on vector models and indexing
Structured CSV files have clear field boundaries. Avoid chunking that breaks the integrity of associated fields such as drug batches and settlement amounts. Semi-structured scanned documents contain OCR recognition errors. Adjust preprocessing logic before vectorization. Frequently updated transaction data requires indexes to support incremental synchronization. Avoid excessively long full reindexing processes. Text with multiple unit measurements must retain the binding relationship between units and corresponding quantities during chunking. Prevent mismatches between units and numerical values during vector matching.
How to set the configuration
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the total field length of individual ledger records in pharmaceutical distribution, avoiding splitting associated fields |
| Overlap Length | 100–150 characters | Retains cross-chunk continuity of key information such as batch numbers and drug names |
defaultToken | 512 | Adapts to the token proportion of individual ledger texts, aligning with the average length of pharmaceutical distribution fields |
maxToken | 4096 | Covers the maximum text length of single-batch ledger summary reports, meeting vector encoding requirements |
| Similarity Threshold | 0.75–0.85 | Distinguishes text features of different drug batches and circulation nodes, avoiding incorrect matching |
| Single Index Limit | First 2000 entries | Controls resource usage for single-batch indexing, adapting to batch processing of 100,000-level data |
Three common mistakes
- Phenomenon: After uploading 100,000 CSV entries to the knowledge base, vectorization takes more than 30 minutes. Cause: Batch indexing configuration is not enabled, and submitting all data at once causes excessive resource usage.
- Phenomenon: After importing a CSV file, the final number of index entries is several thousand fewer than the original file. Cause: Some entries contain unrecognizable special characters such as symbols in drug approval numbers, and are filtered out during the preprocessing stage.
- Phenomenon: Vector scores are identical after Docker deployment, but local runs work normally. Cause: OCR model cache files are not correctly mounted inside the container, and all texts use default feature encoding with no differentiated vectors generated.
How to confirm the configuration is correct
- Upload a single ledger test entry containing special characters, and verify that all key fields and unit information are retained in the generated vector results.
- Check indexing task logs to confirm that the batch size of batch indexing matches the configured single index limit, with no abnormal interruptions.
- Compare vector encoding results between local and Docker deployments to confirm that OCR models and word segmentation configurations are fully synchronized.
- Trigger an incremental indexing task, and verify that new entries are correctly added to the existing index set.
The values provided here are common starting points, and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.