apache/hadoop · error · IOException

tried to deserialize {} bytes of data! newLength must be no

Error message

tried to deserialize {} bytes of data!  newLength must be non-negative.

What it means

Text.readFields(DataInput, int maxLength) first reads the VInt-encoded length of the encoded bytes. Text.write never emits a negative length, so a negative VInt can only come from a damaged stream or from reading bytes that were not written by Text's layout; the guard fails fast with IOException("tried to deserialize N bytes of data! newLength must be non-negative.") instead of attempting a huge allocation.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java:355

  @Override
  public String toString() {
    try {
      return decode(bytes, 0, length);
    } catch (CharacterCodingException e) { 
      throw new RuntimeException("Should not have happened", e);
    }
  }

  @Override
  public void readFields(DataInput in) throws IOException {
    int newLength = WritableUtils.readVInt(in);
    readWithKnownLength(in, newLength);
  }

  public void readFields(DataInput in, int maxLength) throws IOException {
    int newLength = WritableUtils.readVInt(in);
    if (newLength < 0) {
      throw new IOException("tried to deserialize " + newLength +
          " bytes of data!  newLength must be non-negative.");
    } else if (newLength >= maxLength) {
      throw new IOException("tried to deserialize " + newLength +
          " bytes of data, but maxLength = " + maxLength);
    }
    readWithKnownLength(in, newLength);
  }

  /**
   * Skips over one Text in the input.
   * @param in input in.
   * @throws IOException raised on errors performing I/O.
   */
  public static void skip(DataInput in) throws IOException {
    int length = WritableUtils.readVInt(in);
    WritableUtils.skipFully(in, length);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Check the producer's write() order mirrors the reader's readFields() order exactly.
2. Verify the stream/file integrity (checksums, `hadoop fs -text`) and re-run the producer if corrupt.
3. Use the maxLength overload on untrusted input so bad lengths fail with a clear message instead of OOM.
4. Log the stream position when it fires to locate which record is damaged.

try {
  text.readFields(in, maxLength);
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("newLength must be non-negative")) {
    throw new IOException("Corrupt stream at " + in + ": negative Text length", e);
  }
  throw e;
}

When it happens

Trigger: Decoding a Text from a corrupt or truncated stream; reading at a wrong stream position (e.g., treating a value region as a Text, or field order drift between a custom write() and readFields()); a hostile/corrupt VInt from untrusted input data.

Common situations: Custom Writable record layouts where write/read field order diverged; corrupted shuffle or HDFS data; parsing bytes that were never Text-serialized; partially written files from killed writers.

Related errors

• tried to deserialize {} bytes of data, but maxLength = {}
• encoded array component type {} is not a candidate primitive
• encoded array length is negative {}
• data was too long to write! Expected less than or equal to
• Exception while get content summary

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1316d1a67ebe101e. Report an issue: GitHub.