apache/hadoop · error · IOException

encoded array length is negative {}

Error message

encoded array length is negative {}

What it means

readFields() reads the array length as a 4-byte int after the component type and rejects negative values with IOException. A negative length is impossible for an array written by write() (which writes Array.getLength of a real array), so this always signals corrupted, truncated, or misaligned input bytes.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ArrayPrimitiveWritable.java:222

   */
  @Override
  public void readFields(DataInput in) throws IOException {
    
    // read and set the component type of the array
    @SuppressWarnings("deprecation")
    String className = UTF8.readString(in);
    Class<?> componentType = getPrimitiveClass(className);
    if (componentType == null) {
      throw new IOException("encoded array component type "
          + className + " is not a candidate primitive type");
    }
    checkDeclaredComponentType(componentType);
    this.componentType = componentType;
  
    // read and set the length of the array
    int length = in.readInt();
    if (length < 0) {
      throw new IOException("encoded array length is negative " + length);
    }
    this.length = length;
    
    // construct and read in the array
    value = Array.newInstance(componentType, length);

    // do the inner loop.  Walk the decision tree only once.
    if (componentType == Boolean.TYPE) {             // boolean
      readBooleanArray(in);
    } else if (componentType == Character.TYPE) {    // char
      readCharArray(in);
    } else if (componentType == Byte.TYPE) {         // byte
      readByteArray(in);
    } else if (componentType == Short.TYPE) {        // short
      readShortArray(in);
    } else if (componentType == Integer.TYPE) {      // int
      readIntArray(in);
    } else if (componentType == Long.TYPE) {         // long

View on GitHub (pinned to 2add963021)

Solutions

1. Enable/verify checksums (CRC) on the storage path and re-read or delete the corrupt file.
2. Check that the reader consumes exactly the bytes the writer produced (same framing, no skipped bytes).
3. Catch IOException per record, log the offset, and skip/quarantine bad records if the job tolerates it.
4. If corruption recurs, inspect the producing job's output and intermediate (spill/shuffle) health.

try {
  w.readFields(in);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("negative")) {
    throw new DataCorruptionException("Stream misaligned or corrupt at " + position, e);
  }
  throw e;
}

When it happens

Trigger: Bytes damaged in transit or on disk; reading from the wrong stream position (off-by-a-few bytes makes the int land on payload); partial write followed by read; checksums disabled so corruption propagates to the application layer.

Common situations: Reading a corrupted HDFS block or local spill file with checksums disabled/ignored; a custom RPC layer that truncates payloads; reading a file written with a different record framing.

Related errors

• encoded array component type {} is not a candidate primitive
• No URI in deserialized Path
• Encoded type {} converted to valid component type {} but no
• Cannot initialize the class: {clazz}
• tried to deserialize {} bytes of data! newLength must be no

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2bb05b0cdb97ccb0. Report an issue: GitHub.