What the data for this category looks like
Data sources include registered or archived formulas, raw material safety documents, weighing and emulsification equipment data, ERP/MES systems, laboratory test results, and packaging material version documents. Updates are triggered by each production batch or each line change. Daily documents include formula work instructions, weighing and feeding records, emulsification or filling records, cleaning and site clearance records, semi-finished and finished product inspection reports, and label review forms. Fields in documents include restricted substance content, emulsification temperature, production batch number, packaging material version number, and others. Units include degrees Celsius, grams, batch numbers, and others.
What constraints these characteristics impose on document parsing and chunking
High-frequency, small-batch document updates require chunking rules that flexibly adapt to the structure of single-batch production documents. Multiple document types, including tables, images, and numeric fields, require parsing tools to preserve format associations and avoid splitting critical business fields. Differences in document formats across sources, such as Word feeding records and PDF archived formulas, require parsing configurations that support multiple file types. Batch submission scenarios require controlling the upper limit of parsing and chunking time to avoid task timeouts. It is also necessary to retain business associations between fields after chunking. For example, place restricted substance content and its corresponding raw material records in the same chunk.
How to configure the settings
This article applies to FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapt to the field density of cosmetics manufacturing documents, avoid splitting critical parameters such as restricted substance content and emulsification temperature from their corresponding descriptions |
chunkSplitter | Chunk by paragraph + heading | Preserve the hierarchical structure of work instructions and inspection records, and ensure complete association between operating steps and supporting parameters |
customPdfParse | Enabled | Accurately extract tables and embedded images from archived formulas and inspection reports, avoid format confusion or content loss |
UPLOAD_FILE_MAX_SIZE | 100 MB | Adapt to the upload requirements of batch submission of multiple line-change production documents, avoid interception of large-volume files |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapt to multi-document submissions during batch line changes, avoid parsing timeouts for large-volume work instructions or batch tasks |
chunkSettingMode | Manual configuration | Adjust chunking rules for different document types such as feeding records and label review forms to match the association requirements of business fields |
Three common mistakes
- After importing inspection records in Markdown format, image links are lost and cannot be displayed normally. Cause: Custom parsing rules are not configured to match embedded device photos and process diagram links in cosmetics manufacturing documents. The default parsing does not retain domain name associations.
- A
net::ERR_INCOMPerror occurs when accessing the login page in a local area network deployment scenario. Cause:PARSE_FILE_TIMEOUT_SECONDSis not adjusted according to local server bandwidth, orUPLOAD_FILE_MAX_SIZEconfiguration exceeds local storage limits. - When calling an API to obtain chunked content, the result for a specified index cannot be located. Cause:
chunkSettingModedoes not enable index tracking functionality, orchunkSizeis set beyond the actual content length of the document, causing chunk index misalignment.
How to confirm the configuration is correct
- Upload a single formula work instruction, check if the parsed text retains core fields such as restricted substance content and emulsification temperature, and if image links carry complete domain names.
- Submit batch cleaning records generated during line changes, confirm that no parsing timeout errors occur, and all documents have completed chunking.
- Call APIs related to chunking, verify that the returned results include clear chunk index fields, and that corresponding content can be located via the index.
- Adjust
chunkSizeto different ranges, compare the integrity of chunked content, and ensure that critical parameters and their descriptive text are not split.
The values listed here are common starting points, and should be evaluated against the user's own samples.