What this category of data looks like
Typical data sources include formula systems, raw material batch records, internal mixers, Mooney viscometers, vulcanizers, and laboratory testing equipment. Daily documents include formula sheets, weighing records, mixing curves, rubber compound quick test reports, vulcanization process cards, mold usage records, and finished product performance reports. Data is updated per truckload of rubber, per batch of raw materials, and per mold production cycle. Document fields include rubber type ratio, weighing value, mixing temperature, vulcanization time, and other parameters. Some documents include process curve charts.
What constraints these characteristics impose on document parsing and chunking
High-frequency updated production data requires parsing processes to have low latency to avoid impacting the timeliness of quality audits. Multiple types of structured documents, including tables and curve charts, require parsing components to support extraction rules for both plain text and embedded charts. This prevents splitting of key process-related information. Fields with specific units require retaining the binding between numerical values and units during chunking, to prevent parameter ambiguity in subsequent retrieval. Batch high-frequency uploaded documents require reasonable configuration of file size and timeout limits, to prevent parsing interruptions that cause loss of production data.
How to set configurations
This chapter’s configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the single-segment parameter length of mixing curves and vulcanization process cards, avoiding splitting key process nodes |
chunkSplitter | Split by paragraph + table header | Matches the structured layout of formula sheets and weighing records, retaining the association between fields and their corresponding values |
customPdfParse | Enable structured parsing | Extracts numerical points from mixing curves and table fields from vulcanization process cards, avoiding loss of format information from plain text parsing |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the file size and parsing complexity of single-batch mixing records, preventing parsing interruptions from timeouts |
UPLOAD_FILE_MAX_SIZE | 500 MB | Supports batch uploading of multiple rubber compound test reports per batch, meeting the update requirements of high-frequency production data |
chunkSettingMode | Manual configuration | Adjusts chunking rules for different document types, such as formula sheets and vulcanization curves, to adapt to parsing needs for specific scenarios |
Three common configuration mistakes
- Phenomenon: When calling the knowledge base creation API, no parsing, ready, or failed status fields are returned. Real-time tracking of parsing progress is not possible. Cause: API parsing progress synchronization configuration is not enabled, and the status return switch is disabled by default.
- Phenomenon: When parsing a document containing an HTTP response message with a Set-Cookie field, this field is not extracted into the chunked content. Cause: Default parsing rules do not cover HTTP response header fields, and no custom parsing rules are configured to match the Set-Cookie identifier.
- Phenomenon: After uploading a PDF document embedded with a mixing curve image, the parsing result does not associate the image with its corresponding process parameters. Cause: PDF structured image parsing configuration is not enabled, and the binding between the image and its context is not retained during chunking.
How to confirm the configuration is correct
- Submit a single mixing curve PDF document, check if the API return result includes parsing status fields to confirm normal status synchronization.
- Upload a sample HTTP response message document containing a Set-Cookie field, verify that the chunked result includes this field content.
- Upload a formula sheet PDF embedded with process curve images, confirm that the parsing result retains the association between the images and their corresponding parameters.
- Batch upload multiple rubber compound test reports, verify that the parsing timeout and file size limit configurations match actual upload requirements.
The values provided are common starting points and should be evaluated against the reader's own samples.