What data for this category looks like
Data for this category comes from customer surface treatment specifications, bath test results, rectifier operation data, online water quality monitoring records, and hazardous waste management ledgers. Update cycles cover per bath run, per shift, and fixed monitoring periods. Daily documents include bath analysis records, process parameter cards, fixture usage records, film thickness and adhesion reports, hazardous waste ledgers, and wastewater monitoring records. Most document fields are process parameters, monitoring values, and compliance records. Units cover concentration, temperature, current, film thickness, weight, and similar categories.
What constraints do these characteristics impose on the document parsing and chunking step
Documents in this category have high update frequencies and diverse types, covering formats such as bath records, process cards, and compliance ledgers. Document parsing must adapt to structured and semi-structured content, and avoid breaking the binding relationship between process parameters and their corresponding monitoring data during chunking. Field units vary across different documents. Chunking must retain the association between values and their units to prevent parameter matching errors during subsequent calls. Records related to hazardous waste and wastewater involve compliance requirements. Chunking must ensure that compliance items are not split across different chunks, to avoid missing compliance information during bidding. Scattered data sources also lead to inconsistent document formatting. Parsing must adapt to non-standard tables to prevent loss of key fields during parsing.
How to set configuration
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents contains multiple sets of continuous process parameters and monitoring data. This range retains the integrity of a single set of data and avoids splitting parameters |
chunkSplitter | Split by paragraph + table | Adapts to mixed-format documents such as bath records and process cards, and retains the association between data within paragraphs and tables |
customPdfParse | Enabled | Most PDF documents in this category are formatted process cards and monitoring reports. Enabling this option retains the original typesetting and field structure |
PARSE_FILE_TIMEOUT_SECONDS | 120 seconds | Some large hazardous waste ledgers or batch monitoring record files have large file sizes. This duration covers the complete parsing process |
chunkSettingMode | Manual configuration | Chunking rules must be adjusted for different document types. Manual configuration allows flexible adaptation to multiple document formats |
Three common mistakes
- Issue: An error occurs during PDF parsing, and the log shows parsing failure. Cause: The
customPdfParseconfiguration is not enabled. The default parsing logic cannot adapt to electroplating process and monitoring PDF documents with complex tables. - Issue: Uploaded files are not sent directly to the large model, but are converted to plain text by the system before being sent. Cause: The
customPdfParseconfiguration is not enabled, triggering the general text parsing process and failing to retain the original document typesetting and field structure. - Issue: In the parsed chunking results, film thickness values and their corresponding units are split into different chunks. Cause: The
chunkSizevalue is too small, causing continuous process parameters and their units to be split, breaking the binding relationship of the data.
How to confirm the configuration is correct
- A typical electroplating process PDF document is uploaded, and the parsed text is checked to confirm whether it retains the binding relationship between fields such as bath concentration and temperature within tables and their units.
- The
customPdfParseparameter is added to the file collection creation API request, and the parsing configuration returned by the interface is confirmed to match the preset value. - The
chunkSizevalue is adjusted, document splitting results across different chunk lengths are compared, and it is confirmed that a single set of process parameters is not split. - A single large hazardous waste ledger document is uploaded, and it is confirmed that the parsing time does not exceed the
PARSE_FILE_TIMEOUT_SECONDSconfiguration value, with no timeout errors.
The values provided here are common starting points, and should be evaluated against applicable samples.