apache/hadoop · error · IllegalArgumentException

tried to release a buffer that was not created by this strea

Error message

tried to release a buffer that was not created by this stream, {}

What it means

DFSInputStream.releaseBuffer(ByteBuffer) handles buffers returned by the zero-copy read API: it removes the buffer from the stream's extendedReadBuffers map, which associates each mmap'd or pool-backed buffer with the resource that must be freed (ClientMmap or ByteBufferPool). A buffer that is not in the map - wrong stream, already released, or never produced by this stream - triggers IllegalArgumentException. The map remove() means each buffer can be released exactly once, by exactly the stream that created it.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1977

      getExtendedReadBuffers().put(buffer, clientMmap);
      readStatistics.addZeroCopyBytes(length);
      DFSClient.LOG.debug("readZeroCopy read {} bytes from offset {} via the "
          + "zero-copy read path.  blockEnd = {}", length, curPos, blockEnd);
      success = true;
    } finally {
      if (!success) {
        IOUtils.closeStream(clientMmap);
      }
    }
    return buffer;
  }

  @Override
  public synchronized void releaseBuffer(ByteBuffer buffer) {
    if (buffer == EMPTY_BUFFER) return;
    Object val = getExtendedReadBuffers().remove(buffer);
    if (val == null) {
      throw new IllegalArgumentException("tried to release a buffer " +
          "that was not created by this stream, " + buffer);
    }
    if (val instanceof ClientMmap) {
      IOUtils.closeStream((ClientMmap)val);
    } else if (val instanceof ByteBufferPool) {
      ((ByteBufferPool)val).putBuffer(buffer);
    }
  }

  @Override
  public synchronized void unbuffer() {
    closeCurrentBlockReaders();
  }

  @Override
  public boolean hasCapability(String capability) {
    switch (StringUtils.toLowerCase(capability)) {
    case StreamCapabilities.READAHEAD:

View on GitHub (pinned to 2add963021)

Solutions

1. Release each buffer exactly once, to the stream that produced it: wrap release in a provenance-tracking map (ByteBuffer -> stream) and remove the entry as you release.
2. In caches, invalidate buffer entries whenever the owning stream is closed or the block re-opened, so stale buffers are never returned later.
3. Make the release path idempotent in your wrapper: only call stream.releaseBuffer(buf) if provenance.remove(buf) returns the owning stream.
4. Audit double-release paths in finally blocks: a finally that releases after the try already released is the classic source.

Example fix

// before
try {
  ByteBuffer b = zeroCopyIn.read(pool, len, opts);
  consume(b);
} finally {
  in.releaseBuffer(bufRef); // throws if consume() already released it
}

// after
try {
  ByteBuffer b = zeroCopyIn.read(pool, len, opts);
  consume(b);
} finally {
  DFSInputStream owner = (DFSInputStream) bufferOwners.remove(bufRef);
  if (owner != null) owner.releaseBuffer(bufRef); // exactly once
}

Map<ByteBuffer, DFSInputStream> owners = new IdentityHashMap<>();

// record provenance when the buffer is produced
ByteBuffer b = dfsIn.read(pool, len, opts);
if (b != null) owners.put(b, dfsIn);

// release only if this stream produced it, exactly once
DFSInputStream owner = owners.remove(buffer);
if (owner != null) {
  owner.releaseBuffer(buffer);
} // foreign or already-released buffers are ignored, not released

try {
  in.releaseBuffer(buf);
} catch (IllegalArgumentException e) {
  if (String.valueOf(e.getMessage()).contains("not created by this stream")) {
    log.warn("buffer double-release or foreign buffer ignored"); // benign cleanup race
  } else throw e;
}

When it happens

Trigger: Calling releaseBuffer() twice with the same ByteBuffer (the first call removes it from the map, the second throws); returning a readahead buffer to a different FSDataInputStream than it was read from - typical when a cache keyed by path/block hands buffers to a re-opened stream; releasing a foreign buffer that never came from this stream.

Common situations: HBase-style readahead/bucket caches that pool zero-copy buffers across region/block reopens; error paths where cleanup releases a buffer that a success path already released; refactored code that moved buffer ownership between classes without moving the release.

Related errors

• can't read a negative number of bytes.
• ${className} does not support enhanced byte buffer access.
• Cannot seek to negative offset
• Stream is closed!
• BlockReader failed to seek to {}. Instead, it seeked to {}.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/27bee6d0651d1d18. Report an issue: GitHub.