Document Parsing and Chunking for Compulsory Education Dropout Prevention: Compare Documents and Locate Differences

Dropout Prevention: review parsing for document checks. Compare the source records, required inputs and downstream results before handing the task over.

What this category's data looks like

For the compulsory education dropout prevention and control material verification and discrepancy localization category, materials come from dropout risk ledgers, persuasion-to-return records, and home visit records, with an update cadence that rolls with the semester and persuasion-to-return milestones. The ledger uses student name, student registration number, dropout risk level, record date, and signature as fields; persuasion-to-return records include the persuasion target, number of persuasion attempts, persuasion date, and receipt signature; home visit records include the home visit target, home visit date, home visit conclusion, and parent signature. When these three types of materials enter the parsing and chunking function, they are first stored separately by file and data row. The materials themselves serve as data rows to be indexed, and the name, quantity, date, and signature fields are split into retrievable fragments for subsequent retrieval and comparison.

Configuration and implementation checks

This stage is executed by the parsing and training task (server-side task), which outputs data rows and index status (parsed, indexing, indexed). The discrepancy list is then completed by retrieval and subsequent processing.

ObjectCheckVerification action
Dropout risk ledgerThe four fields of name, quantity, date, and signature are consistent between the two materialsAfter uploading the ledger, check whether the data row status is parsed, then check the indexing and indexed transitions, and use chunkSize and chunkSplitter to verify whether the chunk boundaries cut off student names and dates
Persuasion-to-return recordsThe persuasion target, number of persuasion attempts, persuasion date, and receipt signature are consistent with the corresponding ledger recordCompare the name and quantity of the same object in the persuasion-to-return records and the ledger, check whether the receipt signature is missing, and verify the processing mode progress corresponding to trainingType
Home visit recordsThe home visit target, home visit date, home visit conclusion, and parent signature are consistent with the ledger and persuasion-to-return recordsLocate the discrepant fields by date and object, confirm whether the home visit record version is synchronized, and check whether the data row status is indexed
File records and data rowsThe identities of file records and data rows are distinguishable from each otherVerify file records and data rows separately, and confirm that the unique data identifier is correctly bound

Three common errors

  • Symptom: newly uploaded materials cannot be found. Cause: the data row is in parsed or indexing, and data to be indexed does not participate in recall, so it is necessary to wait for indexed.
  • Symptom: empty data rows remain after training fails. Cause: after content parsing, it is immediately stored as a visible data row, and data from failed training can be edited or deleted.
  • Symptom: the name, quantity, and date of the same record do not match between the two materials. Cause: the version is not synchronized or the signature receipt is missing, and the fields are not aligned after chunking.

How to verify the setup

This function's acceptance: upload one material, observe the data row status transitions and the coverage scope of the rebuild task, confirm that the three states parsed, indexing, and indexed are consistent with the interface, and that changing the model only rebuilds data whose indexing has already been completed.

Business test: use one real record to find fields inconsistent with the ledger, and verify the four items of name, quantity, date, and signature.

  • Open the data row to verify the status, and confirm that it is in one of the three states and consistent with the interface display
  • Look at the data rows covered by the rebuild task, and confirm that it only covers data whose indexing has already been completed