Document Parsing and Chunking for Undergraduate Curriculum Planning: Find Supporting Evidence

Program Revision: review parsing for evidence lookup. Compare the source records, required inputs and downstream results before handing the task over.

What this category's data looks like

The revision of undergraduate program plans is based on three types of materials: program plans, course syllabi, and credit structure tables. These materials are sourced from the university's various schools and departments and from academic affairs archives, and are updated by academic year or revision batch. When these three types of materials enter this function, they play different roles: the program plan serves as the overarching framework, the course syllabus provides course-level evidence, and the credit structure table carries quantitative and structural fields. Field characteristics are concentrated in course names, credit amounts, course offering dates, signatures, and receipts. When the names, quantities, or dates of the same record do not match across two materials, when versions are not synchronized, or when signatures or receipts are missing, these are the most common obstacles encountered when querying the business basis before revision. After materials enter the parsing and training tasks, visible data rows are first formed, and then they enter the three-state index flow, for retrieval and subsequent processing to refer back to the original text.

Configuration and implementation checks

The parsing and training task (server-side task) outputs only data rows and index status (parsed, indexing, indexed). Evidence entries that can be traced back to the original text shall be handled subsequently by retrieval and downstream processing. Upon completion of parsing, data rows are immediately persisted and bound to a unique data identifier. The index transitions through three states, and data pending indexing does not participate in recall.

ObjectCriteriaVerification Action
Training programCourse name, credit count, course start date, and signed receipt can be traced back to the original text, and it is the current versionOpen the data row to verify status and version, and compare course name, count, date, and signature item by item
Course syllabusCourse name, credit count, course start date, and signed receipt can be traced back to the original text, and it is the current versionSeparately verify the document record and the data row to confirm that the syllabus fields are consistent with the training program
Credit structure tableCredit count, date, and signed receipt can be traced back to the original text, and it is the current versionReview the data rows covered by the reconstruction task to confirm that the count and date are consistent between the two materials
Data row statusIt is in one of the three states and is consistent with the interface displayOpen the data row to verify the status and confirm that parsed, indexing, and indexed are consistent with the interface

Three common errors

  • Symptom: A newly uploaded material cannot be found in search. Cause: The data row is still in a pending-indexing state and is inherently isolated by the retrieval layer; successful parsing does not equal a retrieval hit.
  • Symptom: The name, quantity, and date of the same record do not match across two materials. Cause: The program of study and the course syllabus versions are not synchronized, and fields have not been compared item by item.
  • Symptom: Empty data rows remain after a failed training run. Cause: Failed data is not edited or deleted in a timely manner, and materials lacking signatures or receipts are mixed into the indexing process.

How to verify the setup

Acceptance for this feature: upload one material, observe the data row status transitions and the coverage scope of the rebuild task, verify that the three states parsed, indexing, and indexed are consistent with the UI, and confirm that the rebuild covers only data that has already completed indexing.

Business test: ask a cross-material evidence question, check whether the answer provides traceable sources, and verify that the tokens of the cited text and the number of array entries are recorded separately.

  • Verify the file record and the data row separately, confirm that the identities of the file record and the data row are distinguishable, and check the processing mode corresponding to trainingType.
  • Examine the data rows covered by the rebuild task, confirm that only data that has already completed indexing is covered, and verify the chunking results corresponding to chunkSize and chunkSplitter.