instructure/canvas-lms · error

AI response appears truncated - the response may have…

Error message

AI response appears truncated - the response may have exceeded length limits. Please try with a shorter prompt or fewer criteria.

What it means

extract_text_from_response extracts content between <tag>...</tag> markers in the LLM response. If an opening <tag> exists but the closing </tag> is missing, the response was cut off (max token/output length exceeded) and the service raises this user-facing truncation error rather than parsing partial criteria.

Solutions

1. Retry with a shorter prompt or fewer criteria per request (the error message's own guidance).
2. Increase the LLM max output token limit in the llm_config used by call_llm_with_prefill.
3. Instruct the model to be more concise (shorter criterion descriptions) to fit within the output limit.
4. Split generation into multiple smaller requests (fewer criteria each).

Example fix

// before
criteria = service.generate_criteria_via_llm(prompt: huge_prompt, criteria_count: 12)
// after
# smaller batch + explicit brevity instruction
prompt << "\nKeep each criterion description under 200 characters."
criteria = service.generate_criteria_via_llm(prompt: prompt, criteria_count: 4)
rescue RuntimeError => e
  raise e unless e.message.include?("AI response appears truncated")
  retry with fewer criteria / higher token limit

# estimate output size before calling
return :too_large if requested_criteria_count > 6 # heuristic cap to avoid truncation

begin
  criteria = service.generate_criteria_via_llm(...)
rescue RuntimeError => e
  raise unless e.message.include?('AI response appears truncated')
  retry_with(fewer_criteria: true, larger_token_limit: true)
end

When it happens

Trigger: LLM response ended mid-XML block because max_output_tokens was hit — typically when generating many criteria, long descriptions, or a long user prompt consuming most of the context/output budget.

Common situations: User asks for a large number of detailed criteria; model configured with a small token limit; provider silently truncates long completions; network/provider streaming cut short leaving malformed XML.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Cannot find criterion with id #
• Cannot regenerate criteria with learning outcomes attached
• Cedar unavailable
• Content exceeds # character limit
• Document index status sync failed for ai_experience #

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/99c23f9af9fdb84d. Report an issue: GitHub.

Appendix: source

Thrown at app/services/rubric_llm_service.rb:794

    JSON.pretty_generate(original)
  end

  # Extract inner text between XML-like tags in an LLM response.
  #
  # Example:
  #   text = "... <RUBRIC_DATA>hello</RUBRIC_DATA> ..."
  #   extract_text_from_response(text, tag: "RUBRIC_DATA") # => "hello"
  #
  # Raises a more specific error if the response appears truncated (opening tag found but no closing tag).
  def extract_text_from_response(response_text, tag:)
    return nil if response_text.blank? || tag.blank?

    regex = %r{<#{Regexp.escape(tag)}>(.*?)</#{Regexp.escape(tag)}>}m
    match = response_text.match(regex)

    if match.nil? && response_text.include?("<#{tag}>")
      Rails.logger.error("Truncated LLM response detected - opening <#{tag}> found but closing </#{tag}> missing")
      raise "AI response appears truncated - the response may have exceeded length limits. Please try with a shorter prompt or fewer criteria."
    end

    match ? match[1].strip : nil
  end

  # Quote a value as a JSON string without the surrounding quotes escaping issues.
  #
  # Example:
  #   escape_value(%{She said "hi"}) # => "\"She said \\\"hi\\\"\""
  #   (and we later strip the outer quotes when re-parsing)
  def escape_value(str)
    return "" if str.nil?

    JSON.generate(str.to_s)[1..-2]
  end

  # Reverse of escape_value – interpret a line value back into plain text.
  # If JSON parsing fails (e.g., malformed escape sequences from LLM), returns the original string.

View on GitHub (pinned to 1c9f0bb801)