What the data for this category looks like
Data for this category comes from concentrate and auxiliary material analysis, DCS systems, cell voltage and current monitoring, laboratory testing, and energy and environmental monitoring. Daily documents include batching lists, smelting records, electrolytic cell condition logs, test reports, anode and cathode quality records, and environmental protection ledgers. These documents are updated in real time for each batch of raw materials, each furnace, or each cell. Most documents include process parameters with specific units, and their structure combines tabular records and continuous operating condition text.
Constraints on Document Parsing and Chunking
The multi-source heterogeneous document structure, real-time update frequency, and specific process parameter requirements of this category create multiple constraints for document parsing and chunking. First, documents contain tabular test data and continuous operating condition text. Parsing must retain the correspondence between fields and values to avoid splitting that breaks process logic. Second, real-time updated records for individual furnaces or cells vary in length. Chunking must adapt to complete operating condition units of different lengths, to avoid merging content across batches or cells. Third, documents from multiple sources have large format differences. Parsing and chunking rules must be flexibly adjusted to match the structural characteristics of different document types.
Configuration Settings
This configuration applies to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category’s documents include continuous operating condition text and process parameters with units. This range retains complete records for a single furnace or cell, avoiding overly short chunks that break process logic or overly long chunks that reduce retrieval accuracy |
chunkSplitter | Custom delimiter + line break | Daily documents are already separated by process steps via line breaks. Using custom delimiters can specify titles such as "batching list" and "smelting record", avoiding merging content across different document types |
customPdfParse | Enabled | Most documents for this category include tabular test data and cell condition parameters. Enabling this option retains table structures and field correspondence, improving parsing completeness |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Single electrolytic cell condition logs or smelting records may include multiple days of operating condition data, leading to long parsing times. This range prevents task timeouts |
chunkSettingMode | Manual configuration | This category has large structural differences across document types. Manual configuration adapts to chunking needs for different documents such as batching lists and environmental protection ledgers |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Some environmental protection ledgers or long-term cell condition records for this category have large file sizes. This upper limit meets conventional upload requirements |
Three Common Configuration Errors
- Symptom: After setting a custom delimiter, chunking still merges multiple paragraphs or splits a single paragraph. Cause: The matching logic between
chunkSizeand overlap length was not adjusted synchronously, orchunkSettingModewas not correctly configured to manual mode. This causes the system’s default chunking rules to override the custom delimiter settings. - Symptom: PDF processing fails with a structured error message after deploying Marker via Docker. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter was not correctly configured, the PDF parsing dependency path in the environment variables was not set, or the uploaded file exceeded theUPLOAD_FILE_MAX_SIZElimit. - Symptom: After chunking, the knowledge base cannot associate the original document source. Cause: The basic configuration for document traceability was not enabled, or document metadata fields were not retained during chunking. This causes each chunk to not be bound to the original file name and upload information.
How to Verify Correct Configuration
- Upload a single batching list PDF, and check if the chunking results retain complete parameters for a single batch of materials, with no cross-paragraph merging or splitting of individual paragraphs.
- Run a parsing task for a single electrolytic cell condition log, and confirm that the task duration falls within the range set for
PARSE_FILE_TIMEOUT_SECONDS, with no timeout errors. - View the chunked document metadata, and confirm that each chunk is associated with the original file name and upload time, allowing traceability to the corresponding document.
- Adjust
chunkSizeto different ranges, compare the completeness of the chunking results, and confirm that they align with the process logic of this category’s documents.
The values provided here are common starting points, and should be evaluated against the user’s own samples.