apache/hadoop · error · IOException

Can't read FileStatusProto with negative size of

Error message

Can't read FileStatusProto with negative size of ${size}

What it means

FileStatus.readFields(DataInput) expects the wire format written by FileStatus.write(DataOutput): a 4-byte int size followed by exactly that many FileStatusProto bytes. If the size int decodes to a negative value the record cannot be valid, so the read aborts with IOException. A negative prefix almost always means the bytes were never in this format, or the stream position is misaligned after an earlier error.

Solutions

1. Confirm writer and reader run the same Hadoop version and that FileStatus.write (int size + protobuf bytes) actually produced the bytes being read
2. If the payload is raw protobuf, skip readFields and use FileStatusProto.parseFrom + PBHelper.convert
3. After any exception on a stream, do not keep reading at the current offset - reopen or resync before the next readFields
4. Hex-dump the first 8 bytes at the read position to verify the length prefix is a small positive int

Example fix

// before
FileStatus st = new FileStatus();
st.readFields(dataIn); // throws: next 4 bytes were not a size prefix

// after: parse raw protobuf from an InputStream instead
FileStatusProto proto = FileStatusProto.parseFrom(inputStream);
FileStatus st = PBHelper.convert(proto);

try {
  status.readFields(in);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("negative")) {
    // stream misaligned or corrupt - do NOT retry on the same stream
    in.close();
    throw new IllegalStateException("Corrupt FileStatus stream at read offset", e);
  }
  throw e;
}

When it happens

Trigger: Invoking new FileStatus().readFields(in) (directly or through Writable-cloning/RPC machinery) on a DataInput whose next 4 bytes are not the length prefix: reading raw FileStatusProto bytes without the int prefix, continuing to read a stream after a prior short read or exception desynchronized the offset, or deserializing data written by an incompatible Hadoop version.

Common situations: Mixed Hadoop client/server versions in one pipeline; custom RPC or cache code hand-reading FileStatus bytes; truncated or partially-written sequence/RPC files; unit tests feeding arbitrary byte arrays into readFields.

Related errors

• Invalid block array length
• absolutePath + " is a file"
• absolutePath + ": No such file or directory."
• adding an additional class would exceed the maximum number…
• Bad status:

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/387ef8d2e3e804f5. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java:501

    sb.append("; hasAcl=" + hasAcl())
        .append("; isEncrypted=" + isEncrypted())
        .append("; isErasureCoded=" + isErasureCoded())
        .append("}");
    return sb.toString();
  }

  /**
   * Read instance encoded as protobuf from stream.
   * @param in Input stream
   * @see PBHelper#convert(FileStatus)
   * @deprecated Use the {@link PBHelper} and protobuf serialization directly.
   */
  @Override
  @Deprecated
  public void readFields(DataInput in) throws IOException {
    int size = in.readInt();
    if (size < 0) {
      throw new IOException("Can't read FileStatusProto with negative " +
          "size of " + size);
    }
    byte[] buf = new byte[size];
    in.readFully(buf);
    FileStatusProto proto = FileStatusProto.parseFrom(buf);
    FileStatus other = PBHelper.convert(proto);
    isdir = other.isDirectory();
    length = other.getLen();
    block_replication = other.getReplication();
    blocksize = other.getBlockSize();
    modification_time = other.getModificationTime();
    access_time = other.getAccessTime();
    setPermission(other.getPermission());
    setOwner(other.getOwner());
    setGroup(other.getGroup());
    setSymlink((other.isSymlink() ? other.getSymlink() : null));
    setPath(other.getPath());
    attr = attributes(other.hasAcl(), other.isEncrypted(),

View on GitHub (pinned to 2add963021)