{"record":{"id":"e724482f1a44d419","repo":"instructure/canvas-lms","slug":"autograde-criteria-count-mismatch-for-submission-submission","errorCode":null,"errorMessage":"[AutoGrade] Criteria count mismatch for submission #{submission.id}: got #{merged_data.length}, expected #{rubric.data.length}","messagePattern":"\\[AutoGrade\\] Criteria count mismatch for submission #(.+?): got #(.+?), expected #(.+?)","errorType":"console","errorClass":"CedarAi::Errors::GraderError","httpStatus":null,"severity":"error","filePath":"app/services/auto_grade_orchestration_service.rb","lineNumber":97,"sourceCode":"    )\n    missing_criteria = get_criteria_missing_grades(auto_grade_result.grade_data, rubric)\n\n    if missing_criteria.any?\n      # filter rubric to only include missing criteria\n      relevant_rubric = rubric.data.select { |item| missing_criteria.include?(item[:description]) }\n\n      grade_data = GradeService.new(\n        assignment: assignment_text,\n        essay: self.class.extract_essay_text(submission),\n        rubric: relevant_rubric,\n        root_account_uuid:,\n        current_user: @current_user\n      ).call\n\n      merged_data = merge_new_grade_data_with_existing(grade_data, auto_grade_result.grade_data || [])\n\n      unless get_criteria_missing_grades(merged_data, rubric).empty?\n        Rails.logger.warn(\"[AutoGrade] Criteria count mismatch for submission #{submission.id}: got #{merged_data.length}, expected #{rubric.data.length}\")\n        raise CedarAi::Errors::GraderError, I18n.t(\"Grading could not be completed. Please try again.\")\n      end\n\n      auto_grade_result.update!(\n        root_account_id: submission.course.root_account_id,\n        grade_data: merged_data,\n        error_message: nil,\n        grading_attempts: auto_grade_result.grading_attempts + 1\n      )\n    end\n\n    auto_grade_result if auto_grade_result.persisted?\n  rescue => e\n    retryable = e.is_a?(CedarAi::Errors::GraderError)\n    handle_grading_failure(\n      error_message: \"Grading failed: #{e.message}\",\n      submission:,\n      auto_grade_result:,","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/instructure/canvas-lms/blob/1c9f0bb8013ed69c4f2efe11fd483025469b7e6c/app/services/auto_grade_orchestration_service.rb#L79-L115","documentation":"AutoGradeOrchestrationService#get_grade_data raises CedarAi::Errors::GraderError when, after merging the auto-grader's grade_data with existing grade data, any rubric criterion is still missing a grade (get_criteria_missing_grades non-empty). The log 'Criteria count mismatch ... got X, expected Y' means the merged grade data does not cover every criterion in the rubric, so grading is aborted with a user-facing I18n error.","triggerScenarios":"run_auto_grader -> get_grade_data where the auto grader (AutoGradeService) returns grade_data that, merged with existing assessment data, leaves one or more rubric criteria ungraded — e.g. LLM grader omitted a criterion, returned fewer/duplicate entries, or rubric.data contains criteria the grader does not recognize.","commonSituations":"Rubric edited after grading started (criteria added/removed); LLM grader output truncated or malformed; criteria with unusual types the auto grader skips; submissions regraded with a stale rubric version.","solutions":["Log/inspect merged_data vs rubric.data to find which criteria are missing grades and why the grader omitted them.","Re-run the auto grade so the LLM grader regenerates complete grade_data covering all criteria.","Verify the rubric used by the submission matches the rubric the grader was built against (rubric.data length and criteria ids).","Add coverage checks before merging (map grader output by criterion id) and fall back to manual grading for criteria the grader cannot score."],"exampleFix":"// before\nunless get_criteria_missing_grades(merged_data, rubric).empty?\n  Rails.logger.warn(\"[AutoGrade] Criteria count mismatch ...\")\n  raise CedarAi::Errors::GraderError, I18n.t(\"Grading could not be completed. Please try again.\")\nend\n\n// after: retry once with explicit missing-criteria prompt before failing\nmissing = get_criteria_missing_grades(merged_data, rubric)\nif missing.any?\n  merged_data = retry_auto_grade_for_criteria(submission, rubric, missing)\n  raise CedarAi::Errors::GraderError, I18n.t(\"Grading could not be completed. Please try again.\") if get_criteria_missing_grades(merged_data, rubric).any?\nend","handlingStrategy":"validation","validationCode":"// before running the grader\nrubric_criteria_ids = rubric.data.map { |d| d[:id] || d['id'] }\nraise ArgumentError, \"rubric has no criteria\" if rubric_criteria_ids.empty?\n# after grader returns, pre-check coverage\nmissing = rubric_criteria_ids - (grade_data.map { |g| g[:criterion_id] || g['criterion_id'] })\nif missing.any?\n  Rails.logger.warn(\"Grader missing criteria: #{missing.inspect}\")\nend","typeGuard":"def complete_grade_data?(grade_data, rubric)\n  expected = rubric.data.map { |d| d['id'].to_s }.sort\n  got = grade_data.map { |g| (g['criterion_id'] || g[:criterion_id]).to_s }.uniq.sort\n  expected == got\nend","tryCatchPattern":"begin\n  orchestration_service.run_auto_grader(submission)\nrescue CedarAi::Errors::GraderError => e\n  FlashMessage.error(e.message)\n  # surface 'Grading could not be completed. Please try again.' to the user\nend","preventionTips":["Always prompt the LLM grader with the full explicit criteria list and require one grade entry per criterion id.","Validate grader output against rubric.data ids immediately after each call and retry before merging.","Detect rubric edits between grading attempts and invalidate stale grade data.","Key grade_data by criterion id rather than array position to survive criteria reordering."],"tags":["ai-grading","rubric","validation","llm-output"],"backgroundTag":"schema-validation-failed","analyzedSha":"1c9f0bb8013ed69c4f2efe11fd483025469b7e6c","analyzedAt":"2026-09-15T20:33:18.891Z","contentChangedAt":"2026-09-15T20:33:18.891Z","schemaVersion":2},"datasetVersion":"2026-09-23T02:17:17.105Z"}