This document is written based on the configuration guidelines for FastGPT v4.17.0.
What this category of data looks like
Daily documents for this category include batching records, coating and calendering parameters, winding and lamination records, liquid injection records, formation and grading data, and OCV and safety test reports. Data sources include raw material batch ledgers, MES systems, slurry and coating testing equipment, dew point monitoring devices, formation and grading cabinets, and laboratory testing data. Documents are generated per production batch or individual cell. Update frequencies cover real-time updates at the batch or individual cell level. Some test reports include structured parameters and numerical results.
What constraints these characteristics impose on document parsing and chunking
Real-time and batch-level update frequencies require the document parsing and chunking process to be efficient. This avoids delaying question-and-answer responses for production links. Documents exported from multiple source devices have inconsistent formats. The structured parameter layout must be accurately preserved to prevent loss of key process values after parsing. Documents have high parameter density. Indicators such as slurry solid content and coating surface density must remain fully associated. Chunking must not destroy the logical integrity of process steps or test items. Electrochemical systems have strict requirements for parameter consistency. Chunking results must ensure that parameters for the same process link are presented together. This avoids information fragmentation caused by cross-chunk queries.
How to set the configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
UPLOAD_FILE_MAX_SIZE | 1000 MB | Matches the total upload volume of documents for a single cell batch, to prevent large file upload interruptions |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses parsing time for complex test reports, to prevent long document parsing timeouts |
customPdfParse | Enable Marker parsing mode | Adapts to PDF formats exported from multiple source devices, preserves structured parameter layouts |
chunkSplitter | Use paragraph + fixed character boundary | Balances parameter integrity and chunk granularity, prevents splitting critical parameter groups |
chunkSize | 800–1200 characters | Matches the parameter density of cell documents, ensures a single chunk contains complete process steps or test items |
chunkSettingMode | Manual configuration | Adjusts parameters for different document types (such as batching records vs formation and grading reports) to fit business scenarios |
Three common mistakes
- Uploading a 3 MB PDF triggers an upload error. The cause is failure to adjust the
UPLOAD_FILE_MAX_SIZEconfiguration, as the default threshold is lower than the file size. - After setting custom delimiters, chunking still merges multiple paragraphs or splits a single paragraph. The cause is that the
chunkSizevalue does not match the document paragraph length, and adjustments were not made based on the combined rules ofchunkSplitter. - Parsing a PDF throws an error after deploying Marker via Docker. The cause is failure to correctly configure environment variables to point to Marker's parsing dependencies, or not extending
PARSE_FILE_TIMEOUT_SECONDSfor parsing timeouts.
How to confirm the configuration is correct
- Upload a single typical document (such as a coating and calendering parameters PDF), check that the upload progress completes without timeout or error prompts.
- Review the chunking results, confirm that key parameters such as coating surface density and dew point value are not split into different chunks, and that paragraph boundaries meet expectations.
- Adjust
chunkSizeto different ranges, compare the content integrity after chunking, and confirm that the value fits the current document type. - Verify that when using Marker parsing mode, tables and parameter lists exported from devices are fully preserved without formatting errors.
The values provided are common starting points and should be measured against the reader's own samples.