ruby/ruby · error · Gem::SafeMarshal::Reader::DataTooShortError

expected #{n} bytes, got #{str.inspect}

Error message

expected #{n} bytes, got #{str.inspect}

What it means

DataTooShortError from SafeMarshal's read_bytes: the IO returned a non-nil string for a declared n-byte read but its bytesize < n, i.e. the stream ends mid-field. Distinct from the EOF case (nil return), this catches data that exists but is shorter than its length prefix claims, a classic sign of corruption in marshaled gem specifications.

Source

Thrown at lib/rubygems/safe_marshal/reader.rb:59

        raise UnconsumedBytesError, "expected EOF, got #{@io.read(10).inspect}... after top-level element #{root.class}" unless @io.eof?
        root
      end

      private

      MARSHAL_VERSION = [Marshal::MAJOR_VERSION, Marshal::MINOR_VERSION].map(&:chr).join.freeze
      private_constant :MARSHAL_VERSION

      def read_header
        v = @io.read(2)
        raise UnsupportedVersionError, "Unsupported marshal version #{v.bytes.map(&:ord).join(".")}, expected #{Marshal::MAJOR_VERSION}.#{Marshal::MINOR_VERSION}" unless v == MARSHAL_VERSION
      end

      def read_bytes(n)
        raise NegativeLengthError if n < 0
        str = @io.read(n)
        raise EOFError, "expected #{n} bytes, got EOF" if str.nil?
        raise DataTooShortError, "expected #{n} bytes, got #{str.inspect}" unless str.bytesize == n
        str
      end

      def read_byte
        @io.getbyte || raise(EOFError, "Unexpected EOF")
      end

      def read_integer
        b = read_byte

        case b
        when 0x00
          0
        when 0x01
          read_byte
        when 0x02
          read_byte | (read_byte << 8)
        when 0x03

View on GitHub (pinned to 0e5b888e1c)

Solutions

1. Regenerate the affected spec/gem: gem pristine <gem>-v <ver> or uninstall/reinstall the gem
2. Compare the file size with a known-good copy (same gem from rubygems.org) to confirm truncation; re-download if sizes differ
3. Audit scripts/tools that rewrite cache files in place; replace in-place writes with atomic write-then-rename
4. Verify no text-mode or encoding transformations touched the binary file in transit

Example fix

# before
str = File.binread(path, mode: 'rb')
Gem::SafeMarshal.safe_load(str) # DataTooShortError

# after
Gem::Specification.from_yaml(File.read(path)) rescue (gem pristine ...) # or simply:
spec = Gem::Package.new(redownloaded_gem).spec

def length_fields_fit?(path)
  # cheap size sanity: compare against a freshly downloaded copy or stored checksum
  digest = Digest::SHA256.file(path).hexdigest
  digest == EXPECTED_DIGESTS[File.basename(path)]
end

begin
  Gem::SafeMarshal.safe_load(data)
rescue Gem::SafeMarshal::Reader::DataTooShortError
  restore_file # delete + re-download / gem pristine
end

When it happens

Trigger: A length-prefixed field (string bytes, symbol name, ivar blob) whose prefix overruns the actual remaining data: e.g. the last string in a .gemspec declares 200 bytes but only 50 exist because the tail was overwritten or never written.

Common situations: Partially flushed cache files after a crash or kill during gem install; disk-full truncation; files mangled by an editor, encoding conversion, or transfer tool (FTP ASCII mode, bad UTF-8 sanitize).

Related errors

• expected #{n} bytes, got EOF
• expected EOF, got #{@io.read(10).inspect}... after top-level
• Unsupported marshal version #{v.bytes.map(&:ord).join(".")},
• expected #{count} elements, but only #{remaining} bytes rema
• Unknown marshal type discriminator #{type.chr.inspect} (#{ty

AI-assisted analysis of ruby/ruby@0e5b888e1c (2026-08-21). Data as JSON: /api/errors/0fd3921f2644b469. Report an issue: GitHub.