What the data for this category looks like
The typical data for this category comes from security operations teams, internal training teams, and external security service providers. Daily documents fall into four categories: training courseware, drill plans, phishing email templates, and drill result reports. Data sources include employee contact lists, email gateways, internal training platforms, and user click statistics systems. Update frequency is mostly quarterly or annual. Document structures include structured drill steps, unstructured email text, and statistical result fields. No fixed unified unit format exists across documents. Some documents contain embedded images or tables.
What constraints these characteristics impose on the document parsing and chunking step
Differences between the four document types require parsing tools to support multiple formats. Training courseware and drill reports mostly use PDF or exported Markdown formats. These require retention of embedded images and typesetting structures. Phishing email templates need complete retention of text context, to avoid splitting that breaks logical messaging flow. Data sources include web links such as publicly shared Yuque documents. These require support for unrestricted web parsing. Low update frequency means parsing processes must be stable and reliable. Single parsing failures must not affect batch tasks. Statistical data fields have no unified format. Original text structures must be retained to support subsequent Q&A.
How to set configurations
The following configurations are based on the parameter logic of FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Documents in this category are mostly training courseware and drill reports. Each chunk must cover complete operational steps or template fragments, to avoid splitting critical information |
chunkSplitter | markdown | Daily documents are mostly PDF-converted Markdown or results from Yuque web parsing. Using Markdown chunking preserves structures like headings and lists |
customPdfParse | Enabled | Training courseware mostly use PDF format. Enabling this extracts embedded images and preserves original typesetting |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Large batch drill reports take longer to parse. This avoids parsing failures caused by timeouts |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Some bulk training courseware packages have large file sizes. This accommodates large file upload requirements |
chunkSettingMode | manual | Chunking rules need adjustment for different document types, including email templates and drill reports, to fit scenario-specific needs |
Three common configuration mistakes
- Symptom: Uploading a PDF training courseware results in missing embedded images in parsed results, or images that cannot be accessed by large models. Cause: The
customPdfParseconfiguration is not enabled. The default parsing process does not extract binary data from embedded images. - Symptom: Entering a publicly shared Yuque link triggers a parsing failure prompt. Cause: The link is not fully publicly accessible, or web parsing adaptation functions are not enabled.
- Symptom: Uploading a phishing email template results in content truncation or broken context in parsed results. Cause: The
chunkSizesetting is too small. This splits complete email template content into multiple independent chunks, losing contextual logical connections.
How to confirm configurations are properly set
- Upload a test PDF training courseware. Check that parsed results retain all embedded images and original typesetting.
- Enter a test public web link. Confirm that the parsing task status is successful and extracted content is complete and free of missing parts.
- Adjust
chunkSizeto different ranges. Upload a test phishing email template document. Confirm that no content truncation occurs. - Upload a drill report larger than standard single file size. Confirm that the parsing task does not trigger a timeout error.
The values provided are common starting points. They should be evaluated against the user's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.