Shopify/liquid · error · Liquid::SyntaxError

Invalid byte sequence in #

Error message

Invalid byte sequence in #{@ss.string.encoding}

What it means

The tokenizer rescues ::ArgumentError 'invalid byte sequence in <encoding>' raised while slicing the source during tokenization and re-raises it as Liquid::SyntaxError with message "Invalid byte sequence in <encoding>". It means the template source contains bytes that are invalid for its encoding, encountered mid-tokenization rather than at parse's initial valid_encoding? check (e.g. byteslice boundaries).

Solutions

1. Sanitize the source before parsing: source.encode(Encoding::UTF_8, invalid: :replace, undef: :replace) or scrub.
2. Ensure any chunking of template content happens on character boundaries, not byte boundaries.
3. Fix the producer of the template so it emits valid UTF-8.

Example fix

// before
Liquid::Template.parse(dirty_source)
// after
Liquid::Template.parse(dirty_source.scrub)

raise Liquid::SyntaxError, 'invalid bytes' unless source.dup.force_encoding(source.encoding).valid_encoding?

def tokenizable_source?(str)
  str.is_a?(String) && str.valid_encoding?
end

begin
  Liquid::Template.parse(source)
rescue Liquid::SyntaxError => e
  raise unless e.message.start_with?('Invalid byte sequence')
  Liquid::Template.parse(source.scrub)
end

When it happens

Trigger: Tokenizing source containing malformed UTF-8 sequences that survive the initial valid_encoding? guard or arise in StringScanner/byteslice operations inside next_text_token.

Common situations: Templates fetched from external systems with corrupted/mixed-encoding bytes; truncated multi-byte characters at chunk boundaries when streaming template fragments.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Invalid byte sequence in #
• invalid byte sequence in #
• errors.syntax.invalid_template_encoding
• Unexpected character #
• invalid base64 provided to base64_decode

AI-assisted analysis of Shopify/liquid@807d45a6b3 (2026-09-08). Data as JSON: /api/errors/d4c069e923275289. Report an issue: GitHub.

Appendix: source

Thrown at lib/liquid/tokenizer.rb:108

      end

      next_text_token
    end

    def next_text_token
      start = @ss.pos

      unless @ss.skip_until(TAG_OR_VARIABLE_START)
        token = @ss.rest
        @ss.terminate
        return token
      end

      pos = @ss.pos -= 2
      @source.byteslice(start, pos - start)
    rescue ::ArgumentError => e
      if e.message == "invalid byte sequence in #{@ss.string.encoding}"
        raise SyntaxError, "Invalid byte sequence in #{@ss.string.encoding}"
      else
        raise
      end
    end

    def next_variable_token
      start = @ss.pos - 2

      byte_a = byte_b = @ss.scan_byte

      while byte_b
        byte_a = @ss.scan_byte while byte_a && byte_a != CLOSE_CURLEY && byte_a != OPEN_CURLEY

        break unless byte_a

        if @ss.eos?
          return byte_a == CLOSE_CURLEY ? @source.byteslice(start, @ss.pos - start) : "{{"
        end

View on GitHub (pinned to 807d45a6b3)