{"record":{"id":"5126b55b68f79102","repo":"Shopify/liquid","slug":"invalid-byte-sequence-in-ss-string-encoding","errorCode":null,"errorMessage":"Invalid byte sequence in #{ss.string.encoding}","messagePattern":"Invalid byte sequence in #(.+?)","errorType":"exception","errorClass":"Liquid::SyntaxError","httpStatus":null,"severity":"error","filePath":"lib/liquid/lexer.rb","lineNumber":166,"sourceCode":"            type, pattern = NEXT_MATCHER_JUMP_TABLE[peeked]\n\n            if type && (t = ss.scan(pattern))\n              # Special case for \"contains\"\n              output << if type == :id && t == \"contains\" && output.last&.first != :dot\n                COMPARISON_CONTAINS\n              else\n                [type, t]\n              end\n            else\n              raise_syntax_error(start_pos, ss)\n            end\n          end\n        end\n        # rubocop:enable Metrics/BlockNesting\n        output << EOS\n      rescue ::ArgumentError => e\n        if e.message == \"invalid byte sequence in #{ss.string.encoding}\"\n          raise SyntaxError, \"Invalid byte sequence in #{ss.string.encoding}\"\n        else\n          raise\n        end\n      end\n\n      def raise_syntax_error(start_pos, ss)\n        ss.pos = start_pos\n        # the character could be a UTF-8 character, use getch to get all the bytes\n        raise SyntaxError, \"Unexpected character #{ss.getch}\"\n      end\n    end\n  end\nend\n","sourceCodeStart":148,"sourceCodeEnd":180,"githubUrl":"https://github.com/Shopify/liquid/blob/807d45a6b3d4568e64e86b375e3702df2c7c860c/lib/liquid/lexer.rb#L148-L180","documentation":"The lexer's StringScanner rescues Ruby's ArgumentError 'invalid byte sequence in <encoding>' and re-raises it as a Liquid::SyntaxError so template parsing fails with a clear message instead of a low-level scanner crash. It means the template source contains bytes that are invalid in the template's declared encoding (usually UTF-8).","triggerScenarios":"Template.parse on source read from a file/network with mixed or corrupted encodings — e.g. UTF-16 bytes, truncated multibyte UTF-8, or Latin-1 bytes force-tagged as UTF-8.","commonSituations":"Downloading templates without setting the HTTP encoding; File.read without an encoding option on non-UTF-8 files; database columns storing binary data; concatenating strings of different encodings before parsing.","solutions":["Re-encode the source before parsing: src.force_encoding('UTF-8').scrub or .encode('UTF-8', invalid: :replace, undef: :replace).","Read files with an explicit encoding: File.read(path, encoding: 'UTF-8').","Set the correct encoding at the source (HTTP response charset, DB client encoding).","Locate the bad bytes with a scrub/validate pass and fix the upstream producer."],"exampleFix":"// before\nLiquid::Template.parse(File.read('tpl.liquid'))  # invalid UTF-8 bytes\n// after\nsrc = File.read('tpl.liquid', encoding: 'UTF-8').scrub\nLiquid::Template.parse(src)","handlingStrategy":"validation","validationCode":"def valid_utf8?(src)\n  src.dup.force_encoding('UTF-8').valid_encoding?\nend","typeGuard":"def scrubbed(src)\n  s = src.dup.force_encoding('UTF-8')\n  s.valid_encoding? ? s : s.scrub\nend","tryCatchPattern":"begin\n  Liquid::Template.parse(src)\nrescue Liquid::SyntaxError => e\n  raise unless e.message.start_with?('Invalid byte sequence')\n  Liquid::Template.parse(scrubbed(src))\nend","preventionTips":["Always read template sources with an explicit encoding.","Set charset correctly on HTTP responses and DB connections storing templates.","Scrub or check valid_encoding? on template content before parsing.","Never concatenate strings of different encodings without re-encoding."],"tags":["ruby","liquid","encoding","utf-8","lexer"],"backgroundTag":"invalid-argument-value","analyzedSha":"807d45a6b3d4568e64e86b375e3702df2c7c860c","analyzedAt":"2026-09-08T11:31:38.917Z","contentChangedAt":"2026-09-08T11:31:38.917Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}