instructure/canvas-lms · info

LLM generated # criteria but expected # . Truncating excess…

Error message

LLM generated #{new_criteria.size} criteria but expected #{desired_criteria_count}. Truncating excess criteria.

What it means

In RubricLlmService#text_to_rubric, after converting LLM output into criteria, the count is validated against desired_criteria_count. If the LLM generated MORE criteria than requested, a warning 'LLM generated N criteria but expected M. Truncating excess criteria.' is logged and the list is truncated with take(). It is a log, not a raise — the mismatch (LLM overshooting the requested rubric size) is silently corrected.

Solutions

1. Accept the warning if truncation is fine — it is expected corrective behavior; verify the truncated rubric is the one you wanted (order matters: take keeps the first N).
2. Strengthen the prompt to state the exact number of criteria required and validate in the LLM response schema.
3. If truncation drops meaningful criteria, log the dropped criteria (new_criteria[desired..]) for debugging and adjust the prompt.
4. Check whether desired_criteria_count passed by the caller matches the user's expectation in the rubric UI.

Example fix

// before
Rails.logger.warn("LLM generated #{new_criteria.size} criteria but expected #{desired_criteria_count}. Truncating excess criteria.")
new_criteria = new_criteria.take(desired_criteria_count)

// after: keep what was dropped visible for debugging
excess = new_criteria.drop(desired_criteria_count)
Rails.logger.warn("Truncating #{excess.size} excess criteria: #{excess.map { |c| c['description'] }}")
new_criteria = new_criteria.take(desired_criteria_count)

// after LLM returns, before text_to_rubric post-processing
raw = JSON.parse(llm_response)
if raw['criteria'].size > desired_criteria_count
  Rails.logger.warn("LLM returned #{raw['criteria'].size} criteria, expected #{desired_criteria_count}; will truncate")
end

def criteria_count_valid?(criteria, desired)
  criteria.is_a?(Array) && criteria.size == desired
end

// Not exception-driven (log-and-truncate). Guard the result instead:
rubric_data = service.public_text_to_rubric(llm_output, desired_criteria_count: 4)
raise "unexpected criteria count" unless rubric_data['criteria'].size <= 4

When it happens

Trigger: text_to_rubric (via public_text_to_rubric or parse_and_transform_regenerated_criteria) receives LLM output whose criteria array is longer than desired_criteria_count — the model ignored or miscounted the requested number of criteria, often when regenerating criteria for an existing rubric.

Common situations: LLM adds extra criteria beyond the requested count; regeneration prompts where desired count comes from the existing rubric but the model invents extras; model version changes that alter output verbosity; prompts that don't strongly constrain the criteria count.

Related errors

• [AutoGrade] Criteria count mismatch for submission #
• Grading could not be completed. Please try again.
• A course did not pass validation
• A # user did not pass validation (user: # , # : # , error…
• A maximum of 50 assessees can be provided at once

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/8b0fbd4ce644d946. Report an issue: GitHub.

Appendix: source

Thrown at app/services/rubric_llm_service.rb:769

          )
          new_criteria << current_crit
        end
        current_crit[field] = value
      elsif type == "rating"
        raise "Rating before criterion" if current_crit.nil?

        rating = current_crit["ratings"].find { |r| r["id"] == raw_id }
        unless rating
          rating = build_blank_rating(id: raw_id, criterion_id: current_crit["id"])
          current_crit["ratings"] << rating
        end
        rating[field] = value
      end
    end

    # Validate criteria count and truncate if necessary
    if new_criteria.size > desired_criteria_count
      Rails.logger.warn("LLM generated #{new_criteria.size} criteria but expected #{desired_criteria_count}. Truncating excess criteria.")
      new_criteria = new_criteria.take(desired_criteria_count)
    elsif new_criteria.size < desired_criteria_count
      raise "Criteria count mismatch: expected #{desired_criteria_count}, got #{new_criteria.size}"
    end

    original["criteria"] = new_criteria
    JSON.pretty_generate(original)
  end

  # Extract inner text between XML-like tags in an LLM response.
  #
  # Example:
  #   text = "... <RUBRIC_DATA>hello</RUBRIC_DATA> ..."
  #   extract_text_from_response(text, tag: "RUBRIC_DATA") # => "hello"
  #
  # Raises a more specific error if the response appears truncated (opening tag found but no closing tag).
  def extract_text_from_response(response_text, tag:)
    return nil if response_text.blank? || tag.blank?

View on GitHub (pinned to 1c9f0bb801)