Shopify/liquid · error · Liquid::SyntaxError

Invalid byte sequence in #

Error message

Invalid byte sequence in #{ss.string.encoding}

What it means

The lexer's StringScanner rescues Ruby's ArgumentError 'invalid byte sequence in <encoding>' and re-raises it as a Liquid::SyntaxError so template parsing fails with a clear message instead of a low-level scanner crash. It means the template source contains bytes that are invalid in the template's declared encoding (usually UTF-8).

Solutions

  1. Re-encode the source before parsing: src.force_encoding('UTF-8').scrub or .encode('UTF-8', invalid: :replace, undef: :replace).
  2. Read files with an explicit encoding: File.read(path, encoding: 'UTF-8').
  3. Set the correct encoding at the source (HTTP response charset, DB client encoding).
  4. Locate the bad bytes with a scrub/validate pass and fix the upstream producer.

Example fix

// before
Liquid::Template.parse(File.read('tpl.liquid'))  # invalid UTF-8 bytes
// after
src = File.read('tpl.liquid', encoding: 'UTF-8').scrub
Liquid::Template.parse(src)
Defensive patterns

Strategy: validation

Validate before calling

def valid_utf8?(src)
  src.dup.force_encoding('UTF-8').valid_encoding?
end

Type guard

def scrubbed(src)
  s = src.dup.force_encoding('UTF-8')
  s.valid_encoding? ? s : s.scrub
end

Try / catch

begin
  Liquid::Template.parse(src)
rescue Liquid::SyntaxError => e
  raise unless e.message.start_with?('Invalid byte sequence')
  Liquid::Template.parse(scrubbed(src))
end

Prevention

When it happens

Trigger: Template.parse on source read from a file/network with mixed or corrupted encodings — e.g. UTF-16 bytes, truncated multibyte UTF-8, or Latin-1 bytes force-tagged as UTF-8.

Common situations: Downloading templates without setting the HTTP encoding; File.read without an encoding option on non-UTF-8 files; database columns storing binary data; concatenating strings of different encodings before parsing.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of Shopify/liquid@807d45a6b3 (2026-09-08). Data as JSON: /api/errors/5126b55b68f79102. Report an issue: GitHub.

Appendix: source

Thrown at lib/liquid/lexer.rb:166

            type, pattern = NEXT_MATCHER_JUMP_TABLE[peeked]

            if type && (t = ss.scan(pattern))
              # Special case for "contains"
              output << if type == :id && t == "contains" && output.last&.first != :dot
                COMPARISON_CONTAINS
              else
                [type, t]
              end
            else
              raise_syntax_error(start_pos, ss)
            end
          end
        end
        # rubocop:enable Metrics/BlockNesting
        output << EOS
      rescue ::ArgumentError => e
        if e.message == "invalid byte sequence in #{ss.string.encoding}"
          raise SyntaxError, "Invalid byte sequence in #{ss.string.encoding}"
        else
          raise
        end
      end

      def raise_syntax_error(start_pos, ss)
        ss.pos = start_pos
        # the character could be a UTF-8 character, use getch to get all the bytes
        raise SyntaxError, "Unexpected character #{ss.getch}"
      end
    end
  end
end

View on GitHub (pinned to 807d45a6b3)