What this category of data looks like
Public transit bus ledger data primarily comes from on-board smart terminals, operation dispatching systems, ticket settlement platforms, and vehicle operation logs. Data is updated daily. A daily operation detail ledger is generated for each single line each day, and a monthly summary analysis report is generated each month. Most documents are structured tables or semi-structured reports. Fields include line number, shift number, departure time, passenger trips, operation mileage, average energy consumption, fault repair records, and more. Units include trips, kilometers, kilowatt-hours, minutes, and others.
What constraints do these characteristics impose on document parsing and chunking?
The multi-scenario structure and update rhythm of public transit bus ledgers impose multiple constraints on document parsing and chunking. A single document ranges from a few lines of daily single-line detail records to dozens of pages of monthly summary reports. Chunking must adapt to documents of different lengths to avoid over-splitting short documents or uneven splitting of long documents. Ledgers contain boolean fields, numeric fields, and text description fields. Parsing must accurately identify data types to avoid type conversion errors. Duplicate headers and cross-page statistical rows exist within documents. Chunking must retain contextual associations to avoid losing the logical correspondence between fields after splitting. Additionally, ledgers generated in batches daily require support for high-concurrency parsing, so reasonable timeout and file size limits must be configured.
How to Configure Settings
These configuration items apply to FastGPT v4.17.0. The specific parameters are as follows:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Each chunk of bus ledger data must hold 1-2 complete shifts of operation data, to avoid splitting the statistical information of a single shift |
chunkSplitter | Chunk by paragraph + semantics | Bus ledgers contain structured tables and text description paragraphs. Chunking by semantics can retain the association between fields |
customPdfParse | Enable structured parsing mode | Most bus ledgers are PDF-format reports. Enabling this mode can accurately extract table fields and types |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Support batch uploading of multi-line ledger documents for monthly summaries, to avoid blocking large documents due to file size limits |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Monthly summary documents have a relatively long length, so sufficient parsing time is required to avoid mid-process timeouts |
chunkSettingMode | Manual configuration | Adjust chunking parameters manually to adapt to scenarios with different ledger document lengths |
Three Common Misconfigurations
- Phenomenon: Parsing failure logs appear after enabling PDF enhanced parsing. Cause: MinerU API access parameters are not configured correctly, causing the custom parsing module to fail to call the third-party parsing service.
- Phenomenon: Boolean fields become null values after being processed by conditional judgment components. Cause: Boolean types were mistakenly converted to string types during parsing, and the original data type was not retained, causing the conditional judgment logic to fail to match.
- Phenomenon: A single monthly summary document is split into more than 3000 chunks, and cannot be added to the knowledge base. Cause: Automatic chunk splitting configuration is not enabled, or the upper limit of the number of chunks per file in the knowledge base is not adjusted.
How to Verify Correct Configuration
- A daily single-line ledger PDF is uploaded, the parsed field list is reviewed, and boolean and numeric fields are confirmed to be correctly identified.
chunkSizeis adjusted to 800 characters, a monthly summary document is uploaded, the chunking preview is reviewed, and each chunk is confirmed to contain complete shift data.- A ledger compressed package larger than 500 MB is uploaded, and the upload and parsing process is confirmed to not be blocked.
- 10 monthly documents are simulated for batch upload, the parsing logs are reviewed, and no timeout or knowledge base entry failure errors are confirmed to have been triggered.
The values given are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.