What this category of data looks like
Data for this category comes from design documents, standard instrument and environmental monitoring data, calibration software, laboratory information systems, and customer range requirements. It is generated and updated each time a device is calibrated. Daily documents include assembly and commissioning sheets, calibration procedures, original calibration records, error curves, metrology certificates, and software version records. Fields include tested part number, calibration date, error value, ambient temperature and humidity, standard instrument number, and other relevant fields. Units include ppm, ℃, kPa, and other metrology-related standard units. Single documents mostly contain the single calibration result for one device. Bulk documents are aggregated records for multiple devices.
What constraints do these characteristics impose on document parsing and chunking?
Since data is updated per device per calibration, documents include a large number of frequently generated small single files and bulk aggregated documents. They also contain non-pure text content such as charts and formulas. Precise matching of document structure is required to retain traceability chain information. Most fields are precise metrology-related numerical values. Chunking must avoid splitting that breaks field associations, to prevent core traceability information in calibration records from being disrupted. Document types vary widely. Structured calibration certificates and unstructured error curve documents are both present. Parsing logic must adapt to different formats to avoid field loss or parsing errors from format mismatches. Frequently generated documents also require stable parsing efficiency to avoid timeouts that disrupt business processes.
Configuration Settings
This configuration applies to FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents contains metrology values and traceability chain information. Chunk length must cover core fields of a single calibration record to avoid breaking field associations |
chunkSplitter | Split by header, footer, and paragraph | Calibration documents often have fixed headers (including laboratory identifiers) and footers (including reviewers). Splitting by paragraph preserves the integrity of individual calibration records |
customPdfParse | Enable chart text extraction mode | This category of documents includes error curves and metrology certificate charts. Enabling this mode extracts labeled text from embedded charts to avoid losing critical data |
PARSE_FILE_TIMEOUT_SECONDS | 60–90 seconds | Single calibration documents contain multiple sets of error data. Sufficient time is needed to complete synchronous extraction of charts and text, to avoid timeout interruptions |
UPLOAD_FILE_MAX_SIZE | 100 MB | Some bulk aggregated calibration documents have large file sizes. This setting covers conventional bulk upload requirements |
chunkSettingMode | Manual configuration | Document formats vary widely in this category. Manual configuration allows precise adjustment of chunking rules to adapt to different types of calibration records and certificates |
Three common configuration errors
- Symptom: Text in image areas of parsed documents is missing, or labeled content from embedded error curves is absent. Cause: The chart extraction mode of
customPdfParseis not enabled. Only pure text is extracted, leading to loss of critical metrology data. - Symptom: After uploading a publicly shared Yuque link, parsing fails with a permission error or "data source not supported" prompt. Cause: The Yuque link is not confirmed to be a publicly accessible link without permission restrictions. FastGPT only supports publicly accessible web pages without access restrictions as knowledge base data sources.
- Symptom: Parsed chunk content includes redundant header and footer information, leading to irrelevant laboratory identifiers or reviewer content appearing during retrieval. Cause:
chunkSplitteris not configured to split by paragraph. Direct fixed-character splitting causes headers and footers to be included in chunks, reducing retrieval accuracy.
How to confirm correct configuration
- Upload a standard metrology certificate PDF. Check that the parsed result includes all metrology fields and error curve labels, to confirm that the
customPdfParsefunction is active. - Adjust
chunkSizeto 800 characters. Upload a single calibration record document. Check that chunks fully retain all core information of a single calibration, with no fields split across chunks. - Upload a bulk calibration aggregated document under 100 MB. Confirm that the parsing task completes within 90 seconds, with no timeout error prompts.
- Test uploading a publicly accessible Yuque document link without permission restrictions. Confirm that parsing succeeds and content is complete, to verify data source adaptation rules.
The values provided are common starting points and should be measured against the reader's own samples.