{"record":{"id":"8b0fbd4ce644d946","repo":"instructure/canvas-lms","slug":"llm-generated-new-criteria-size-criteria-but-expected","errorCode":null,"errorMessage":"LLM generated #{new_criteria.size} criteria but expected #{desired_criteria_count}. Truncating excess criteria.","messagePattern":"LLM generated #(.+?) criteria but expected #(.+?)\\. Truncating excess criteria\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"info","filePath":"app/services/rubric_llm_service.rb","lineNumber":769,"sourceCode":"          )\n          new_criteria << current_crit\n        end\n        current_crit[field] = value\n      elsif type == \"rating\"\n        raise \"Rating before criterion\" if current_crit.nil?\n\n        rating = current_crit[\"ratings\"].find { |r| r[\"id\"] == raw_id }\n        unless rating\n          rating = build_blank_rating(id: raw_id, criterion_id: current_crit[\"id\"])\n          current_crit[\"ratings\"] << rating\n        end\n        rating[field] = value\n      end\n    end\n\n    # Validate criteria count and truncate if necessary\n    if new_criteria.size > desired_criteria_count\n      Rails.logger.warn(\"LLM generated #{new_criteria.size} criteria but expected #{desired_criteria_count}. Truncating excess criteria.\")\n      new_criteria = new_criteria.take(desired_criteria_count)\n    elsif new_criteria.size < desired_criteria_count\n      raise \"Criteria count mismatch: expected #{desired_criteria_count}, got #{new_criteria.size}\"\n    end\n\n    original[\"criteria\"] = new_criteria\n    JSON.pretty_generate(original)\n  end\n\n  # Extract inner text between XML-like tags in an LLM response.\n  #\n  # Example:\n  #   text = \"... <RUBRIC_DATA>hello</RUBRIC_DATA> ...\"\n  #   extract_text_from_response(text, tag: \"RUBRIC_DATA\") # => \"hello\"\n  #\n  # Raises a more specific error if the response appears truncated (opening tag found but no closing tag).\n  def extract_text_from_response(response_text, tag:)\n    return nil if response_text.blank? || tag.blank?","sourceCodeStart":751,"sourceCodeEnd":787,"githubUrl":"https://github.com/instructure/canvas-lms/blob/1c9f0bb8013ed69c4f2efe11fd483025469b7e6c/app/services/rubric_llm_service.rb#L751-L787","documentation":"In RubricLlmService#text_to_rubric, after converting LLM output into criteria, the count is validated against desired_criteria_count. If the LLM generated MORE criteria than requested, a warning 'LLM generated N criteria but expected M. Truncating excess criteria.' is logged and the list is truncated with take(). It is a log, not a raise — the mismatch (LLM overshooting the requested rubric size) is silently corrected.","triggerScenarios":"text_to_rubric (via public_text_to_rubric or parse_and_transform_regenerated_criteria) receives LLM output whose criteria array is longer than desired_criteria_count — the model ignored or miscounted the requested number of criteria, often when regenerating criteria for an existing rubric.","commonSituations":"LLM adds extra criteria beyond the requested count; regeneration prompts where desired count comes from the existing rubric but the model invents extras; model version changes that alter output verbosity; prompts that don't strongly constrain the criteria count.","solutions":["Accept the warning if truncation is fine — it is expected corrective behavior; verify the truncated rubric is the one you wanted (order matters: take keeps the first N).","Strengthen the prompt to state the exact number of criteria required and validate in the LLM response schema.","If truncation drops meaningful criteria, log the dropped criteria (new_criteria[desired..]) for debugging and adjust the prompt.","Check whether desired_criteria_count passed by the caller matches the user's expectation in the rubric UI."],"exampleFix":"// before\nRails.logger.warn(\"LLM generated #{new_criteria.size} criteria but expected #{desired_criteria_count}. Truncating excess criteria.\")\nnew_criteria = new_criteria.take(desired_criteria_count)\n\n// after: keep what was dropped visible for debugging\nexcess = new_criteria.drop(desired_criteria_count)\nRails.logger.warn(\"Truncating #{excess.size} excess criteria: #{excess.map { |c| c['description'] }}\")\nnew_criteria = new_criteria.take(desired_criteria_count)","handlingStrategy":"validation","validationCode":"// after LLM returns, before text_to_rubric post-processing\nraw = JSON.parse(llm_response)\nif raw['criteria'].size > desired_criteria_count\n  Rails.logger.warn(\"LLM returned #{raw['criteria'].size} criteria, expected #{desired_criteria_count}; will truncate\")\nend","typeGuard":"def criteria_count_valid?(criteria, desired)\n  criteria.is_a?(Array) && criteria.size == desired\nend","tryCatchPattern":"// Not exception-driven (log-and-truncate). Guard the result instead:\nrubric_data = service.public_text_to_rubric(llm_output, desired_criteria_count: 4)\nraise \"unexpected criteria count\" unless rubric_data['criteria'].size <= 4","preventionTips":["State the exact criteria count in the prompt and repeat it in the output schema instructions.","Use structured/JSON output mode so the model adheres to requested array sizes.","Keep rubric order stable — take() silently keeps only the first N criteria.","Log dropped criteria so truncation is auditable."],"tags":["llm-output","rubric","truncation","validation"],"backgroundTag":"shape-mismatch","analyzedSha":"1c9f0bb8013ed69c4f2efe11fd483025469b7e6c","analyzedAt":"2026-09-15T20:33:18.891Z","contentChangedAt":"2026-09-15T20:33:18.891Z","schemaVersion":2},"datasetVersion":"2026-09-23T02:17:17.105Z"}