Document Parsing and Chunking for Sino-Foreign Education Establishment and Annual Reports: Compare Documents and Locate Differences

Joint Education: review parsing for document checks. Compare the source records, required inputs and downstream results before handing the task over.

What this category's data looks like

The materials for the establishment and annual reports of Chinese-foreign cooperative education institutions mainly consist of the establishment application, cooperation agreement, and articles of association, supplemented by annual report receipts and signature pages. The establishment application is drafted by the Chinese sponsoring entity, the cooperation agreement is signed jointly by both parties, the articles of association are reviewed by the decision-making body, and the annual report is submitted on an academic-year cycle. After the materials enter this feature, the establishment application serves as the field baseline, the cooperation agreement and articles of association serve as comparison sources, and the annual report receipt is used to confirm the submission status. Field characteristics are concentrated in institution name, number of majors, enrollment scale, signing date, signature and seal, and version number. In different materials, the same record often shows mixed use of abbreviated and full names, different counting bases, dates spanning years, unsynchronized versions, and missing signatures or receipts, so discrepancies must be located field by field.

Configuration and implementation checks

This step is executed by parsing and training tasks (server-side tasks), outputting data rows and index status (parsed, indexing, indexed). The discrepancy list is then completed by retrieval and subsequent processing. After parsing is completed, parsed chunks are immediately persisted as visible data rows and bound to a unique data identifier, and the index transitions through the three states parsed, indexing, and indexed.

ObjectCheckVerification action
Establishment applicationCheck whether the institution name, number of majors, enrollment scale, signing date, and signature and seal are consistent with the cooperation agreementCompare the application and the cooperation agreement field by field, mark discrepancies in names, quantities, and dates, and check whether the signature page and receipt are complete
Cooperation agreementCheck whether the names of both parties, cooperation term, major setup, signing date, and signature and seal are consistent with the articles of associationCheck the file records and data rows separately, confirm that trainingType corresponds to the material type, and record the version number and signing date
Articles of associationCheck whether the institution name, composition of the decision-making body, number of majors, and effective date are consistent with the establishment applicationCheck the clauses of the articles of association against the statements in the application, mark discrepancies in quantities and dates, and confirm whether the versions are synchronized
Data rowIs in one of the three states parsed, indexing, or indexed and consistent with the interface displayOpen the data row to check the status, confirm that chunkSize and chunkSplitter correspond to the material chunking, and check whether the index status affects recall

Three common errors

  • Symptom: newly uploaded materials cannot be found. Cause: the data row is in the parsed or indexing state, and data pending indexing does not participate in recall, so you must wait for indexing to complete.
  • Symptom: the name, quantity, and date of the same record do not match across two materials. Cause: mixed use of abbreviated and full names, different statistical bases, signing dates spanning years, and unsynchronized versions.
  • Symptom: after training fails, empty data rows are left behind or rebuilding takes abnormally long. Cause: failed training data is not cleaned up, changing the model rebuilds only data that has already been indexed, and the scope judgment is incorrect.

How to verify the setup

Acceptance for this feature: upload one material, observe the data row status transitions and the coverage scope of the rebuild task, and confirm that the three-state transitions are consistent with the index status.

Business test: use one real record to find fields inconsistent with the ledger, mark discrepancies in names, quantities, and dates, and submit them to the reviewer for confirmation and write-back.

  • Check the file records and data row identifiers, confirm that identities can be distinguished, and check that trainingType corresponds to the material type
  • Review the data rows covered by the rebuild task, confirm that only data that has already been indexed is covered, and record the discrepancy fields and write them back