apache/hadoop · error · IllegalArgumentException

tried to release a buffer that was not created by this…

Error message

tried to release a buffer that was not created by this stream, {}

What it means

DFSInputStream.releaseBuffer(ByteBuffer) handles buffers returned by the zero-copy read API: it removes the buffer from the stream's extendedReadBuffers map, which associates each mmap'd or pool-backed buffer with the resource that must be freed (ClientMmap or ByteBufferPool). A buffer that is not in the map - wrong stream, already released, or never produced by this stream - triggers IllegalArgumentException. The map remove() means each buffer can be released exactly once, by exactly the stream that created it.

Solutions

1. Release each buffer exactly once, to the stream that produced it: wrap release in a provenance-tracking map (ByteBuffer -> stream) and remove the entry as you release.
2. In caches, invalidate buffer entries whenever the owning stream is closed or the block re-opened, so stale buffers are never returned later.
3. Make the release path idempotent in your wrapper: only call stream.releaseBuffer(buf) if provenance.remove(buf) returns the owning stream.
4. Audit double-release paths in finally blocks: a finally that releases after the try already released is the classic source.

Example fix

// before
try {
  ByteBuffer b = zeroCopyIn.read(pool, len, opts);
  consume(b);
} finally {
  in.releaseBuffer(bufRef); // throws if consume() already released it
}

// after
try {
  ByteBuffer b = zeroCopyIn.read(pool, len, opts);
  consume(b);
} finally {
  DFSInputStream owner = (DFSInputStream) bufferOwners.remove(bufRef);
  if (owner != null) owner.releaseBuffer(bufRef); // exactly once
}

Map<ByteBuffer, DFSInputStream> owners = new IdentityHashMap<>();

// record provenance when the buffer is produced
ByteBuffer b = dfsIn.read(pool, len, opts);
if (b != null) owners.put(b, dfsIn);

// release only if this stream produced it, exactly once
DFSInputStream owner = owners.remove(buffer);
if (owner != null) {
  owner.releaseBuffer(buffer);
} // foreign or already-released buffers are ignored, not released

try {
  in.releaseBuffer(buf);
} catch (IllegalArgumentException e) {
  if (String.valueOf(e.getMessage()).contains("not created by this stream")) {
    log.warn("buffer double-release or foreign buffer ignored"); // benign cleanup race
  } else throw e;
}

When it happens

Trigger: Calling releaseBuffer() twice with the same ByteBuffer (the first call removes it from the map, the second throws); returning a readahead buffer to a different FSDataInputStream than it was read from - typical when a cache keyed by path/block hands buffers to a re-opened stream; releasing a foreign buffer that never came from this stream.

Common situations: HBase-style readahead/bucket caches that pool zero-copy buffers across region/block reopens; error paths where cleanup releases a buffer that a success path already released; refactored code that moved buffer ownership between classes without moving the release.

Related errors

• can't read a negative number of bytes.
• Attempted to read past end of file
• BlockReader failed to seek to
• Cannot seek after EOF
• Cannot seek to negative offset

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/27bee6d0651d1d18. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1977

      getExtendedReadBuffers().put(buffer, clientMmap);
      readStatistics.addZeroCopyBytes(length);
      DFSClient.LOG.debug("readZeroCopy read {} bytes from offset {} via the "
          + "zero-copy read path.  blockEnd = {}", length, curPos, blockEnd);
      success = true;
    } finally {
      if (!success) {
        IOUtils.closeStream(clientMmap);
      }
    }
    return buffer;
  }

  @Override
  public synchronized void releaseBuffer(ByteBuffer buffer) {
    if (buffer == EMPTY_BUFFER) return;
    Object val = getExtendedReadBuffers().remove(buffer);
    if (val == null) {
      throw new IllegalArgumentException("tried to release a buffer " +
          "that was not created by this stream, " + buffer);
    }
    if (val instanceof ClientMmap) {
      IOUtils.closeStream((ClientMmap)val);
    } else if (val instanceof ByteBufferPool) {
      ((ByteBufferPool)val).putBuffer(buffer);
    }
  }

  @Override
  public synchronized void unbuffer() {
    closeCurrentBlockReaders();
  }

  @Override
  public boolean hasCapability(String capability) {
    switch (StringUtils.toLowerCase(capability)) {
    case StreamCapabilities.READAHEAD:

View on GitHub (pinned to 2add963021)