{"record":{"id":"d4c069e923275289","repo":"Shopify/liquid","slug":"invalid-byte-sequence-in-ss-string-encoding-d4c069","errorCode":null,"errorMessage":"Invalid byte sequence in #{@ss.string.encoding}","messagePattern":"Invalid byte sequence in #(.+?)","errorType":"exception","errorClass":"Liquid::SyntaxError","httpStatus":null,"severity":"error","filePath":"lib/liquid/tokenizer.rb","lineNumber":108,"sourceCode":"      end\n\n      next_text_token\n    end\n\n    def next_text_token\n      start = @ss.pos\n\n      unless @ss.skip_until(TAG_OR_VARIABLE_START)\n        token = @ss.rest\n        @ss.terminate\n        return token\n      end\n\n      pos = @ss.pos -= 2\n      @source.byteslice(start, pos - start)\n    rescue ::ArgumentError => e\n      if e.message == \"invalid byte sequence in #{@ss.string.encoding}\"\n        raise SyntaxError, \"Invalid byte sequence in #{@ss.string.encoding}\"\n      else\n        raise\n      end\n    end\n\n    def next_variable_token\n      start = @ss.pos - 2\n\n      byte_a = byte_b = @ss.scan_byte\n\n      while byte_b\n        byte_a = @ss.scan_byte while byte_a && byte_a != CLOSE_CURLEY && byte_a != OPEN_CURLEY\n\n        break unless byte_a\n\n        if @ss.eos?\n          return byte_a == CLOSE_CURLEY ? @source.byteslice(start, @ss.pos - start) : \"{{\"\n        end","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/Shopify/liquid/blob/807d45a6b3d4568e64e86b375e3702df2c7c860c/lib/liquid/tokenizer.rb#L90-L126","documentation":"The tokenizer rescues ::ArgumentError 'invalid byte sequence in <encoding>' raised while slicing the source during tokenization and re-raises it as Liquid::SyntaxError with message \"Invalid byte sequence in <encoding>\". It means the template source contains bytes that are invalid for its encoding, encountered mid-tokenization rather than at parse's initial valid_encoding? check (e.g. byteslice boundaries).","triggerScenarios":"Tokenizing source containing malformed UTF-8 sequences that survive the initial valid_encoding? guard or arise in StringScanner/byteslice operations inside next_text_token.","commonSituations":"Templates fetched from external systems with corrupted/mixed-encoding bytes; truncated multi-byte characters at chunk boundaries when streaming template fragments.","solutions":["Sanitize the source before parsing: source.encode(Encoding::UTF_8, invalid: :replace, undef: :replace) or scrub.","Ensure any chunking of template content happens on character boundaries, not byte boundaries.","Fix the producer of the template so it emits valid UTF-8."],"exampleFix":"// before\nLiquid::Template.parse(dirty_source)\n// after\nLiquid::Template.parse(dirty_source.scrub)","handlingStrategy":"try-catch","validationCode":"raise Liquid::SyntaxError, 'invalid bytes' unless source.dup.force_encoding(source.encoding).valid_encoding?","typeGuard":"def tokenizable_source?(str)\n  str.is_a?(String) && str.valid_encoding?\nend","tryCatchPattern":"begin\n  Liquid::Template.parse(source)\nrescue Liquid::SyntaxError => e\n  raise unless e.message.start_with?('Invalid byte sequence')\n  Liquid::Template.parse(source.scrub)\nend","preventionTips":["Scrub incoming template bytes before parsing","Avoid splitting templates on byte boundaries mid-character","Enforce UTF-8 at every ingestion point"],"tags":["liquid","encoding","tokenizer","utf-8"],"backgroundTag":"invalid-argument-format","analyzedSha":"807d45a6b3d4568e64e86b375e3702df2c7c860c","analyzedAt":"2026-09-08T11:31:38.917Z","contentChangedAt":"2026-09-08T11:31:38.917Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}