apache/hadoop · error · NegativeArraySizeException

Corrupted data: negative string length {length}

Error message

Corrupted data: negative string length {length}

What it means

Thrown by TFile's Utils.readString(DataInput, int): the method reads a VInt length prefix where -1 encodes null, so any other negative value is impossible for a real string and means the underlying bytes are corrupt. Hadoop deliberately throws NegativeArraySizeException (documented on the method) instead of attempting new byte[negativeLength], which would throw the same exception with no context. It signals stream corruption or a mispositioned reader, never a transient condition.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java:301

   * lengths that are -2 or less. or larger than the supplied bound before any
   * buffer is allocated.
   * A length of -1 means "no data" and mapped to a null string.
   *
   * @param in The input stream.
   * @param maxLength The largest permitted encoded length in bytes, negative for no limit.
   * @return The string or null.
   * @throws EOFException input data length exceeds {@code maxLength}.
   * @throws IOException IO failure.
   * @throws NegativeArraySizeException string length was minus two or less.
   */
  public static String readString(DataInput in, int maxLength)
      throws IOException {
    int length = readVInt(in);
    if (length == -1) {
      return null;
    }
    if (length < 0) {
      throw new NegativeArraySizeException("Corrupted data: negative string length "
          + length);
    }
    if (maxLength >= 0 && length > maxLength) {
      throw new EOFException("String length " + length
          + " exceeds the limit of " + maxLength);
    }
    byte[] buffer = new byte[length];
    in.readFully(buffer);
    return Text.decode(buffer);
  }

  /**
   * A generic Version class. We suggest applications built on top of TFile use
   * this class to maintain version information in their meta blocks.
   * 
   * A version number consists of a major version and a minor version. The
   * suggested usage of major and minor version number is to increment major
   * version number when the new storage format is not backward compatible, and

View on GitHub (pinned to 2add963021)

Solutions

1. Treat the input as corrupt: re-transfer or regenerate the file from source data, verifying checksums (HDFS CRC / distcp verification) along the way.
2. Confirm reader positioning: the DataInput must sit exactly where the writer wrote the string; audit offset/length arithmetic that advanced the stream before readString.
3. Reproduce with a minimal reader to find the first bad offset; if the file came from an interrupted write, it is simply incomplete and must be re-produced.
4. Check that writer and reader use the same TFile format version and the same string encoding (Utils.writeString vs raw Text).

try {
  String s = Utils.readString(in, maxLength);
} catch (NegativeArraySizeException e) {
  // stream is corrupt: quarantine the file; do NOT retry the same bytes
  throw new IOException("Corrupt TFile input near offset " + inOffset, e);
}

When it happens

Trigger: Calling Utils.readString — directly or via TFile Reader meta-block parsing (comparator names, version strings written with Utils.writeString) — on a DataInput positioned at bytes that are not a length-prefixed string; reading a TFile that was truncated, partially written, mis-transferred, or written by an incompatible format/version.

Common situations: TFile/sequence output from a killed or crashed job read as if complete; block/key offset math off by a few bytes so the reader lands mid-record; files copied without checksum verification picking up bit corruption; mixing TFile format versions between writer and reader.

Related errors

• Key length out of range: {klen}
• Cannot find matching key in block.
• First key entry size out of range: {size}
• First key length out of range: {firstKeyLength}
• Index entry size out of range: {size}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7e96b5d75734975c. Report an issue: GitHub.