Document Parsing and Chunking for Academic Misconduct Reports and Investigations: Compare Documents and Locate Differences

Research Misconduct: review parsing for document checks. Compare the source records, required inputs and downstream results before handing the task over.

What this category's data looks like

This page addresses the scenario of investigating and handling academic misconduct reports, focusing on the verification and discrepancy localization of three types of materials: report materials, investigation reports, and expert appraisal opinions. Material sources include the original records submitted by the reporter, the interim reports formed by the investigation team, and the appraisal opinions issued by experts; the update cadence iterates multiple times as the investigation progresses. After the three types of materials enter the document parsing and chunking feature, each assumes a different role: report materials serve as the baseline for verification, investigation reports serve as the process record, and expert appraisal opinions serve as the basis for conclusions. Field characteristics center on names, quantities, dates, signatures, and receipts; the same record often fails to match across different materials due to unsynchronized versions or data-entry discrepancies. After chunking, each data row is bound to a unique data identifier, facilitating field-by-field comparison and discrepancy localization.

Configuration and implementation checks

This step is executed by the parsing and training task (server-side task), which outputs data rows and index status (parsed, indexing, indexed). The discrepancy list is then completed by retrieval and subsequent processing. After parsing is complete, data rows are immediately persisted and bound to a unique data identifier; data pending indexing does not participate in recall.

ObjectCheckVerification action
Report materialsWhether names, quantities, and dates are consistent with the investigation reportVerify the material fields against the ledger, and confirm whether the chunking results under chunkSize and chunkSplitter cover all fields
Investigation reportWhether dates, signatures, and receipts are completeCompare the report date with the expert appraisal opinion date, and check whether signatures and receipts are missing
Expert appraisal opinionWhether the conclusion fields are consistent with the investigation report's conclusionVerify the names and quantities in the appraisal opinion, and confirm whether the chunks corresponding to trainingType are complete
Data row statusIs in one of the three states parsed, indexing, indexed and consistent with the interfaceOpen the data row to verify the status, and confirm that data pending indexing has not been recalled

Three common errors

  • Symptom: The quantities for the same record do not match between the report materials and the investigation report. Cause: The two materials were entered separately, versions were not synchronized, and after chunking the fields are scattered across different data rows.
  • Symptom: The investigation report is missing a signature or receipt, making verification impossible. Cause: The material circulation step did not write back, and the persisted data row lacks the corresponding fields.
  • Symptom: A newly uploaded material cannot be found. Cause: The data row is in a pending-indexing state, indexing is not complete, and the retrieval layer naturally isolates it from recall.

How to verify the setup

Feature acceptance: Upload one material, observe the data row status transition from parsed to indexing to indexed, and verify that the rebuild task covers only data that has completed indexing.

Business test: Use one real record to find fields inconsistent with the ledger, hand them to the reviewer for confirmation, and write back.

  • Verify the identity identifiers of the file record and the data row separately, and confirm that the two are distinguishable.
  • Check the data rows covered by the rebuild task, and confirm that it targets only data that has completed indexing.