instructure/canvas-lms · error · CedarAi::Errors::GraderError

[AutoGrade] Criteria count mismatch for submission #

Error message

[AutoGrade] Criteria count mismatch for submission #{submission.id}: got #{merged_data.length}, expected #{rubric.data.length}

What it means

AutoGradeOrchestrationService#get_grade_data raises CedarAi::Errors::GraderError when, after merging the auto-grader's grade_data with existing grade data, any rubric criterion is still missing a grade (get_criteria_missing_grades non-empty). The log 'Criteria count mismatch ... got X, expected Y' means the merged grade data does not cover every criterion in the rubric, so grading is aborted with a user-facing I18n error.

Solutions

1. Log/inspect merged_data vs rubric.data to find which criteria are missing grades and why the grader omitted them.
2. Re-run the auto grade so the LLM grader regenerates complete grade_data covering all criteria.
3. Verify the rubric used by the submission matches the rubric the grader was built against (rubric.data length and criteria ids).
4. Add coverage checks before merging (map grader output by criterion id) and fall back to manual grading for criteria the grader cannot score.

Example fix

// before
unless get_criteria_missing_grades(merged_data, rubric).empty?
  Rails.logger.warn("[AutoGrade] Criteria count mismatch ...")
  raise CedarAi::Errors::GraderError, I18n.t("Grading could not be completed. Please try again.")
end

// after: retry once with explicit missing-criteria prompt before failing
missing = get_criteria_missing_grades(merged_data, rubric)
if missing.any?
  merged_data = retry_auto_grade_for_criteria(submission, rubric, missing)
  raise CedarAi::Errors::GraderError, I18n.t("Grading could not be completed. Please try again.") if get_criteria_missing_grades(merged_data, rubric).any?
end

// before running the grader
rubric_criteria_ids = rubric.data.map { |d| d[:id] || d['id'] }
raise ArgumentError, "rubric has no criteria" if rubric_criteria_ids.empty?
# after grader returns, pre-check coverage
missing = rubric_criteria_ids - (grade_data.map { |g| g[:criterion_id] || g['criterion_id'] })
if missing.any?
  Rails.logger.warn("Grader missing criteria: #{missing.inspect}")
end

def complete_grade_data?(grade_data, rubric)
  expected = rubric.data.map { |d| d['id'].to_s }.sort
  got = grade_data.map { |g| (g['criterion_id'] || g[:criterion_id]).to_s }.uniq.sort
  expected == got
end

begin
  orchestration_service.run_auto_grader(submission)
rescue CedarAi::Errors::GraderError => e
  FlashMessage.error(e.message)
  # surface 'Grading could not be completed. Please try again.' to the user
end

When it happens

Trigger: run_auto_grader -> get_grade_data where the auto grader (AutoGradeService) returns grade_data that, merged with existing assessment data, leaves one or more rubric criteria ungraded — e.g. LLM grader omitted a criterion, returned fewer/duplicate entries, or rubric.data contains criteria the grader does not recognize.

Common situations: Rubric edited after grading started (criteria added/removed); LLM grader output truncated or malformed; criteria with unusual types the auto grader skips; submissions regraded with a stale rubric version.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Grading could not be completed. Please try again.
• LLM generated # criteria but expected # . Truncating excess…
• A course did not pass validation
• A # user did not pass validation (user: # , # : # , error…
• A maximum of 50 assessees can be provided at once

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/e724482f1a44d419. Report an issue: GitHub.

Appendix: source

Thrown at app/services/auto_grade_orchestration_service.rb:97

    )
    missing_criteria = get_criteria_missing_grades(auto_grade_result.grade_data, rubric)

    if missing_criteria.any?
      # filter rubric to only include missing criteria
      relevant_rubric = rubric.data.select { |item| missing_criteria.include?(item[:description]) }

      grade_data = GradeService.new(
        assignment: assignment_text,
        essay: self.class.extract_essay_text(submission),
        rubric: relevant_rubric,
        root_account_uuid:,
        current_user: @current_user
      ).call

      merged_data = merge_new_grade_data_with_existing(grade_data, auto_grade_result.grade_data || [])

      unless get_criteria_missing_grades(merged_data, rubric).empty?
        Rails.logger.warn("[AutoGrade] Criteria count mismatch for submission #{submission.id}: got #{merged_data.length}, expected #{rubric.data.length}")
        raise CedarAi::Errors::GraderError, I18n.t("Grading could not be completed. Please try again.")
      end

      auto_grade_result.update!(
        root_account_id: submission.course.root_account_id,
        grade_data: merged_data,
        error_message: nil,
        grading_attempts: auto_grade_result.grading_attempts + 1
      )
    end

    auto_grade_result if auto_grade_result.persisted?
  rescue => e
    retryable = e.is_a?(CedarAi::Errors::GraderError)
    handle_grading_failure(
      error_message: "Grading failed: #{e.message}",
      submission:,
      auto_grade_result:,

View on GitHub (pinned to 1c9f0bb801)