apache/hadoop · error · IOException

Invalid size: {} for file metadata object

Error message

Invalid size: {} for file metadata object

What it means

SequenceFile.Metadata.readFields throws this IOException when the metadata entry-count field read from the file header is negative. The writer always writes a non-negative count, so a negative value means the bytes being parsed are not a valid sequence-file metadata block: the file is truncated, corrupted, or not a SequenceFile at all. It typically escapes from the SequenceFile.Reader constructor while the header is parsed.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:774

      return new TreeMap<Text, Text>(this.theMetadata);
    }
    
    @Override
    public void write(DataOutput out) throws IOException {
      out.writeInt(this.theMetadata.size());
      Iterator<Map.Entry<Text, Text>> iter =
        this.theMetadata.entrySet().iterator();
      while (iter.hasNext()) {
        Map.Entry<Text, Text> en = iter.next();
        en.getKey().write(out);
        en.getValue().write(out);
      }
    }

    @Override
    public void readFields(DataInput in) throws IOException {
      int sz = in.readInt();
      if (sz < 0) throw new IOException("Invalid size: " + sz + " for file metadata object");
      this.theMetadata = new TreeMap<Text, Text>();
      for (int i = 0; i < sz; i++) {
        Text key = new Text();
        Text val = new Text();
        key.readFields(in);
        val.readFields(in);
        this.theMetadata.put(key, val);
      }    
    }

    @Override
    public boolean equals(Object other) {
      if (other == null) {
        return false;
      }
      if (other.getClass() != this.getClass()) {
        return false;
      } else {

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm it is a sequence file: hadoop fs -cat /path | head -c 4 must show the 'SEQ' magic bytes
2. Verify completeness against the source (compare sizes, hadoop fs -checksum) and re-copy if it differs
3. Delete or regenerate the corrupt/truncated file from its source data
4. Pin writer and reader jobs to the same Hadoop version to avoid header-layout skew

Example fix

// before
SequenceFile.Reader r = new SequenceFile.Reader(conf, Reader.file(p)); // Invalid size IOException

// after: reject non-sequence files before the Reader parses the header
try (FSDataInputStream in = p.getFileSystem(conf).open(p)) {
  byte[] magic = new byte[3];
  in.readFully(magic);
  if (!new String(magic).equals("SEQ")) throw new IOException(p + " is not a SequenceFile");
}
SequenceFile.Reader r = new SequenceFile.Reader(conf, Reader.file(p));

try (FSDataInputStream in = fs.open(p)) {
  byte[] magic = new byte[3];
  in.readFully(magic);
  if (!"SEQ".equals(new String(magic))) {
    throw new IOException(p + " is not a sequence file (bad magic)");
  }
}

try {
  reader = new SequenceFile.Reader(conf, Reader.file(p));
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Invalid size")) {
    // header corruption: quarantine the file, do not retry the same bytes
    quarantine(p);
  } else { throw e; }
}

When it happens

Trigger: Constructing SequenceFile.Reader on a file whose metadata section is cut off (truncated transfer), that is 0 bytes or garbage, that was written by an incompatible tool/version, or whose header bytes were damaged.

Common situations: Interrupted copyToLocal/putFromLocal leaving partial files; a job pointed at a text/Avro file where a .seq path was expected; bad disks or bit rot on archival data; reading files produced by an old Hadoop version with a different header layout.

Related errors

• Can't read FileStatusProto with negative size of ${size}
• Illegal buffer length " + len
• Stream data required
• file or stream must be specified
• file modifier options not compatible with stream

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f3fdd89881e03006. Report an issue: GitHub.