instructure/canvas-lms · error · JSON::ParserError

The AI response was not in the expected format. Please try…

Error message

The AI response was not in the expected format. Please try again.

What it means

JSON::ParserError re-raised by RubricLlmService#parse_and_transform_generated_criteria when the LLM's response for generated rubric criteria cannot be parsed as JSON. The service logs the raw parse failure and re-raises with a user-friendly message asking the caller to retry.

Solutions

1. Retry the generation request — the message is explicitly designed to be retried
2. Log/inspect the raw LLM response to see what invalid output was returned
3. Strip markdown fences / leading text before JSON.parse, or ask the model for strict JSON-only output
4. Reduce criteria_count or total_points so the response is not truncated by token limits

Example fix

// before
raw = llm_client.complete(prompt)
criteria = JSON.parse(raw)
// after
raw = llm_client.complete(prompt)
json = raw.sub(/\A```(?:json)?\s*/m, '').sub(/\s*```\z/m, '')
criteria = JSON.parse(json)

null

null

begin
  criteria = service.generate_criteria_via_llm(...)
rescue JSON::ParserError
  # retry up to N times with backoff; surface 'please try again' to the user
  retry_count += 1
  retry if retry_count < 3
  raise
end

When it happens

Trigger: Calling generate_criteria_via_llm when the model returns non-JSON output: markdown-fenced text, prose preamble, truncated response, or an empty/error completion.

Common situations: LLM provider returning an error message instead of generated content; max_tokens truncating the JSON; prompt changes causing the model to add explanation text around the JSON; rate-limit fallback bodies.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

• Unsupported prompt
• Artifact required for assessing
• Assessment type required for assessing
• assessor and assessee required
• Assessor required for assessing

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/f707decb31f38a6c. Report an issue: GitHub.

Appendix: source

Thrown at app/services/rubric_llm_service.rb:272

    }
  end

  def parse_and_transform_generated_criteria(response, generate_options)
    json_str = "{" + response
    last_index = json_str.rindex("}")
    json_str = json_str[0..last_index] unless last_index.nil?
    ai_rubric = JSON.parse(json_str, symbolize_names: true)

    criteria_count = ai_rubric[:criteria].length
    total_points = generate_options[:total_points].to_f
    points_per_criterion = calculate_points_per_criterion(total_points, criteria_count)

    ai_rubric[:criteria].each_with_index.map do |criterion_data, index|
      build_criterion_from_llm(criterion_data, points_per_criterion[index], generate_options[:use_range])
    end
  rescue JSON::ParserError => e
    Rails.logger.error("Failed to parse LLM response as JSON during generation: #{e.message}")
    raise JSON::ParserError, "The AI response was not in the expected format. Please try again."
  end

  # Calculate points per criterion based on total_points and criteria_count
  def calculate_points_per_criterion(total_points, criteria_count)
    points_per_criterion = (total_points / criteria_count).round(ROUNDING_PRECISION)
    points_for_criterion = {}
    running_total = 0.0

    Array(1..criteria_count).each_with_index do |_, index|
      if index == criteria_count - 1
        points_for_criterion[index] = (total_points - running_total).round(ROUNDING_PRECISION)
      else
        running_total += points_per_criterion
        points_for_criterion[index] = points_per_criterion
      end
    end
    points_for_criterion
  end

View on GitHub (pinned to 1c9f0bb801)