What Data for This Category Looks Like
Data for this category comes from the molten iron and scrap steel data system, L2 process control platform, laboratory, MES system, energy medium system, and equipment status monitoring module. Daily documents include heat plans, batching lists, converter smelting records, composition reports, continuous casting process records, and slab judgment sheets. They cover fields such as heat number, composition proportion, process temperature, casting speed, and quality judgment result, with units including degrees Celsius, percentage, meters per minute, and others. Data update frequencies cover three dimensions: per heat, real-time, and per shift. A single document may contain complete process and inspection information for one or multiple heats.
What Constraints These Characteristics Impose on the "Document Parsing and Chunking" Link
Multi-source heterogeneous data sources require parsing to support business documents in different formats, including structured reports and long-text process descriptions. The per-heat and real-time update characteristics require chunking to match process unit granularity, avoiding splitting critical heat-level process logic. Fields with specific units must be fully retained, otherwise the accuracy of subsequent inspection and audit will be affected. Large volumes of archived historical documents require adaptation to large-file parsing and chunking workflows, while also avoiding loss of critical quality judgment information due to improper chunking rules.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Business documents contain complete heat-level process parameters and judgment results. Chunking must cover the minimum retrievable process unit, avoiding splitting critical parameter groups |
chunkSplitter | Mixed paragraph + fixed-length | There are both structured report paragraphs and long-text process descriptions. Mixed chunking balances logical integrity and retrieval granularity |
customPdfParse | Enabled | Most business documents use standardized report formats. Custom parsing can accurately extract table fields and unit information, avoiding missing critical content |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Single batch documents may contain historical data for multiple heats. Parsing takes a long time, so the setting must accommodate batch processing time requirements |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports batch import of historical smelting record archive packages to meet training needs for full business data |
chunkSettingMode | Manual configuration | Chunking rules must be adjusted for different document types such as composition reports and continuous casting records, to adapt to specific business retrieval logic |
Three Common Misconfigurations
- Phenomenon: Data processing is empty after uploading a PDF, and no content is returned in search tests. Cause:
customPdfParseis not enabled, so the table structure of standardized steel industry reports cannot be recognized, resulting in empty extracted content. - Phenomenon: Data processing is empty during training after local deployment, while the conversation function operates normally. Cause:
PARSE_FILE_TIMEOUT_SECONDSis set too short, and batch historical smelting records are terminated before parsing is completed. - Phenomenon: Some parts of PDFs are recognized, while others are not. Cause: Custom parsing rules are not configured for process parameters with specific units, causing fields containing special units to be misjudged as invalid content.
How to Verify Proper Configuration
- Log in to the FastGPT v4.17.0 backend, upload a single converter smelting record PDF, check the parsed text preview, and confirm that the heat number, carbon content value and unit are fully extracted.
- Check the configuration value of
PARSE_FILE_TIMEOUT_SECONDSto ensure it is greater than the actual measured parsing time of the largest single document. - Test batch uploading multiple business documents of different types, confirm that all documents show a parsing completed status, and there are no empty data entries.
- Perform a search test, enter a query term containing specific process parameters, and confirm that the returned results contain the corresponding chunked content.
The values provided are common starting points and should be measured against the reader's own samples.