apache/hadoop · error · IOException

Can't read FileStatusProto with negative size of ${size}

Error message

Can't read FileStatusProto with negative size of ${size}

What it means

FileStatus.readFields(DataInput) expects the wire format written by FileStatus.write(DataOutput): a 4-byte int size followed by exactly that many FileStatusProto bytes. If the size int decodes to a negative value the record cannot be valid, so the read aborts with IOException. A negative prefix almost always means the bytes were never in this format, or the stream position is misaligned after an earlier error.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java:501

    sb.append("; hasAcl=" + hasAcl())
        .append("; isEncrypted=" + isEncrypted())
        .append("; isErasureCoded=" + isErasureCoded())
        .append("}");
    return sb.toString();
  }

  /**
   * Read instance encoded as protobuf from stream.
   * @param in Input stream
   * @see PBHelper#convert(FileStatus)
   * @deprecated Use the {@link PBHelper} and protobuf serialization directly.
   */
  @Override
  @Deprecated
  public void readFields(DataInput in) throws IOException {
    int size = in.readInt();
    if (size < 0) {
      throw new IOException("Can't read FileStatusProto with negative " +
          "size of " + size);
    }
    byte[] buf = new byte[size];
    in.readFully(buf);
    FileStatusProto proto = FileStatusProto.parseFrom(buf);
    FileStatus other = PBHelper.convert(proto);
    isdir = other.isDirectory();
    length = other.getLen();
    block_replication = other.getReplication();
    blocksize = other.getBlockSize();
    modification_time = other.getModificationTime();
    access_time = other.getAccessTime();
    setPermission(other.getPermission());
    setOwner(other.getOwner());
    setGroup(other.getGroup());
    setSymlink((other.isSymlink() ? other.getSymlink() : null));
    setPath(other.getPath());
    attr = attributes(other.hasAcl(), other.isEncrypted(),

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm writer and reader run the same Hadoop version and that FileStatus.write (int size + protobuf bytes) actually produced the bytes being read
2. If the payload is raw protobuf, skip readFields and use FileStatusProto.parseFrom + PBHelper.convert
3. After any exception on a stream, do not keep reading at the current offset - reopen or resync before the next readFields
4. Hex-dump the first 8 bytes at the read position to verify the length prefix is a small positive int

Example fix

// before
FileStatus st = new FileStatus();
st.readFields(dataIn); // throws: next 4 bytes were not a size prefix

// after: parse raw protobuf from an InputStream instead
FileStatusProto proto = FileStatusProto.parseFrom(inputStream);
FileStatus st = PBHelper.convert(proto);

try {
  status.readFields(in);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("negative")) {
    // stream misaligned or corrupt - do NOT retry on the same stream
    in.close();
    throw new IllegalStateException("Corrupt FileStatus stream at read offset", e);
  }
  throw e;
}

When it happens

Trigger: Invoking new FileStatus().readFields(in) (directly or through Writable-cloning/RPC machinery) on a DataInput whose next 4 bytes are not the length prefix: reading raw FileStatusProto bytes without the int prefix, continuing to read a stream after a prior short read or exception desynchronized the offset, or deserializing data written by an incompatible Hadoop version.

Common situations: Mixed Hadoop client/server versions in one pipeline; custom RPC or cache code hand-reading FileStatus bytes; truncated or partially-written sequence/RPC files; unit tests feeding arbitrary byte arrays into readFields.

Related errors

• Invalid block array length: {}
• parent + " is a file"
• Parent directory doesn't exist: " + parent
• absolutePath + ": No such file or directory."
• Exception while get content summary

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/387ef8d2e3e804f5. Report an issue: GitHub.