apache/hadoop · error · IllegalArgumentException

can't read a negative number of bytes.

Error message

can't read a negative number of bytes.

What it means

DFSInputStream.read(ByteBufferPool, int maxLength, EnumSet<ReadOption>) - the enhanced/zero-copy read entry point - rejects maxLength < 0 with IllegalArgumentException before doing anything (maxLength == 0 returns a shared empty buffer). Callers reach this via ByteBufferUtil.fallbackRead or by calling the enhanced read API directly, e.g. FSDataInputStream.read(ByteBufferPool, int, EnumSet) implementations in frameworks. A negative length is always a caller-side arithmetic bug; no cluster state can produce it.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1865

    }
    closeCurrentBlockReaders();
  }

  /**
   * The immutable empty buffer we return when we reach EOF when doing a
   * zero-copy read.
   */
  private static final ByteBuffer EMPTY_BUFFER =
      ByteBuffer.allocateDirect(0).asReadOnlyBuffer();

  @Override
  public synchronized ByteBuffer read(ByteBufferPool bufferPool,
      int maxLength, EnumSet<ReadOption> opts)
          throws IOException, UnsupportedOperationException {
    if (maxLength == 0) {
      return EMPTY_BUFFER;
    } else if (maxLength < 0) {
      throw new IllegalArgumentException("can't read a negative " +
          "number of bytes.");
    }
    if ((blockReader == null) || (blockEnd == -1)) {
      if (pos >= getFileLength()) {
        return null;
      }
      /*
       * If we don't have a blockReader, or the one we have has no more bytes
       * left to read, we call seekToBlockSource to get a new blockReader and
       * recalculate blockEnd.  Note that we assume we're not at EOF here
       * (we check this above).
       */
      if ((!seekToBlockSource(pos)) || (blockReader == null)) {
        throw new IOException("failed to allocate new BlockReader " +
            "at position " + pos);
      }
    }
    ByteBuffer buffer = null;

View on GitHub (pinned to 2add963021)

Solutions

1. Clamp the computed length: int len = (int) Math.max(0, Math.min(limit - pos, cap)); before calling the enhanced read.
2. Find the subtraction producing the negative value (usually endOffset - curOffset or budget - consumed) and validate its operands.
3. Return early on len == 0 - the API treats it as a valid no-op returning the empty buffer, so a zero-length chunk needs no special casing.

Example fix

// before
int len = (int) Math.min(endOffset - curPos, maxChunk);
in = enhancedIn.read(bufferPool, len, opts); // len < 0 when curPos > endOffset

// after
int len = (int) Math.min(endOffset - curPos, maxChunk);
if (len < 0) len = 0; // or break the loop when curPos >= endOffset
in = enhancedIn.read(bufferPool, len, opts);

int maxLength = (int) Math.min(limit - curPos, maxChunk);
if (maxLength < 0) maxLength = 0; // API treats 0 as a valid no-op returning the empty buffer
ByteBuffer b = in.read(bufferPool, maxLength, opts);

When it happens

Trigger: Passing a negative maxLength computed as a subtraction that underflowed, e.g. (int)(limit - position) where position > limit, or (available() - headerSize) where headerSize > available(). Also direct calls like in.read(pool, -1, opts).

Common situations: Custom zero-copy consumers porting from Hadoop 2.x APIs; chunked transfer loops that size the next read as (endOffset - streamPos) without clamping; unit tests exercising boundary sizes that accidentally feed 0-length pools with negative budgets.

Related errors

• Cannot seek to negative offset
• Mark/reset not supported
• tried to release a buffer that was not created by this strea
• Invalid value: bytesPerChecksum = {} <= 0
• Invalid values: dfs.bytes-per-checksum (={}) must divide blo

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/41617c7244759d1f. Report an issue: GitHub.