{"record":{"id":"3465f1cf450be3e7","repo":"instructure/canvas-lms","slug":"not-a-valid-utf-8-string-string","errorCode":null,"errorMessage":"Not a valid utf-8 string: %{string}","messagePattern":"Not a valid utf-8 string: %(.+?)","errorType":"validation","errorClass":"ParseError","httpStatus":null,"severity":"error","filePath":"lib/outcomes/csv_importer.rb","lineNumber":132,"sourceCode":"\n    def test_header_i18n\n      header = @file.readline\n      has_bom = header.start_with?((+\"\\xEF\\xBB\\xBF\").force_encoding(\"ASCII-8BIT\"))\n      @file.rewind\n      @file.read(3) if has_bom\n      (header.count(\";\") > header.count(\",\")) ? \";\" : \",\"\n    end\n\n    def file_line_count\n      count = @file.each.inject(0) { |c, _line| c + 1 }\n      @file.rewind\n      count\n    end\n\n    def check_encoding(str)\n      encoded = str&.force_encoding(\"utf-8\")\n      valid = (encoded || \"\").valid_encoding?\n      raise ParseError, I18n.t(\"Not a valid utf-8 string: %{string}\", string: str.inspect) unless valid\n\n      encoded\n    end\n\n    def validate_headers(row, _index)\n      main_columns_end = row.find_index(\"ratings\") || row.length\n      headers = row.slice(0, main_columns_end).map(&:to_sym)\n\n      after_ratings = row[(main_columns_end + 1)..] || []\n      after_ratings = after_ratings.compact_blank.map(&:to_s)\n      raise ParseError, I18n.t(\"Invalid fields after ratings: %{fields}\", fields: after_ratings.inspect) unless after_ratings.empty?\n\n      missing = (REQUIRED_FIELDS - headers).map(&:to_s)\n      raise ParseError, I18n.t(\"Missing required fields: %{fields}\", fields: missing.inspect) unless missing.empty?\n\n      invalid = (headers - OPTIONAL_FIELDS - REQUIRED_FIELDS).map(&:to_s)\n      raise ParseError, I18n.t(\"Invalid fields: %{fields}\", fields: invalid.inspect) unless invalid.empty?\n","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/instructure/canvas-lms/blob/1c9f0bb8013ed69c4f2efe11fd483025469b7e6c/lib/outcomes/csv_importer.rb#L114-L150","documentation":"Raised by Outcomes::CsvImporter#check_encoding when a string read from the CSV file is not valid UTF-8. The importer force-encodes the input to UTF-8 and calls valid_encoding?; if the byte sequence cannot be a valid UTF-8 string (e.g. Latin-1 or Windows-1252 bytes), it aborts the parse with a ParseError. It exists to fail fast before rows are parsed with an unrepresentable encoding.","triggerScenarios":"Passing a CSV file whose bytes are not valid UTF-8 (e.g. saved as ISO-8859-1/Windows-1252 by Excel) into the outcome CSV import; embedding a smart quote or accented character in a non-UTF-8 codepage; reading the file with binary/ASCII-8BIT encoding and handing it to check_encoding directly.","commonSituations":"Spreadsheets exported from Excel on Windows in 'CSV (MS-DOS)' or ANSI format; files edited in editors using legacy encodings; downloads from legacy systems using ISO-8859-1.","solutions":["Re-save/re-encode the CSV as UTF-8 (e.g. `iconv -f WINDOWS-1252 -t UTF-8 input.csv > output.csv` or export as 'CSV UTF-8' from Excel).","Detect the actual encoding (`file -i input.csv` or chardet) and convert before importing.","If converting in code, read the file, force_encoding to the known source encoding, then encode('UTF-8')."],"exampleFix":"// before\nCsvImporter.new(File.read('outcomes.csv')) # raw Windows-1252 bytes\n// after\nraw = File.read('outcomes.csv', encoding: 'Windows-1252')\nutf8 = raw.encode('UTF-8')\nCsvImporter.new(utf8)","handlingStrategy":"validation","validationCode":"def utf8?(str)\n  s = str&.force_encoding(\"utf-8\")\n  (s || \"\").valid_encoding?\nend\nraise \"file is not UTF-8\" unless utf8?(File.read(path))","typeGuard":"def valid_utf8?(str)\n  str.is_a?(String) && str.dup.force_encoding(\"UTF-8\").valid_encoding?\nend","tryCatchPattern":"begin\n  importer.parse_file\nrescue Outcomes::CsvImporter::ParseError => e\n  logger.warn(\"CSV encoding problem: #{e.message}\")\n  # re-encode the file and retry once\nend","preventionTips":["Always export CSVs as 'CSV UTF-8' from Excel","Run `file -i` or a chardet check on files before import","Normalize all inbound files with a single encoding-conversion utility"],"tags":["csv","encoding","utf-8","import"],"backgroundTag":"invalid-argument-format","analyzedSha":"1c9f0bb8013ed69c4f2efe11fd483025469b7e6c","analyzedAt":"2026-09-15T20:33:18.891Z","contentChangedAt":"2026-09-15T20:33:18.891Z","schemaVersion":2},"datasetVersion":"2026-09-23T02:17:17.105Z"}