{"record":{"id":"27bee6d0651d1d18","repo":"apache/hadoop","slug":"tried-to-release-a-buffer-that-was-not-created-by-27bee6","errorCode":null,"errorMessage":"tried to release a buffer that was not created by this stream, {}","messagePattern":"tried to release a buffer that was not created by this stream, (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":1977,"sourceCode":"      getExtendedReadBuffers().put(buffer, clientMmap);\n      readStatistics.addZeroCopyBytes(length);\n      DFSClient.LOG.debug(\"readZeroCopy read {} bytes from offset {} via the \"\n          + \"zero-copy read path.  blockEnd = {}\", length, curPos, blockEnd);\n      success = true;\n    } finally {\n      if (!success) {\n        IOUtils.closeStream(clientMmap);\n      }\n    }\n    return buffer;\n  }\n\n  @Override\n  public synchronized void releaseBuffer(ByteBuffer buffer) {\n    if (buffer == EMPTY_BUFFER) return;\n    Object val = getExtendedReadBuffers().remove(buffer);\n    if (val == null) {\n      throw new IllegalArgumentException(\"tried to release a buffer \" +\n          \"that was not created by this stream, \" + buffer);\n    }\n    if (val instanceof ClientMmap) {\n      IOUtils.closeStream((ClientMmap)val);\n    } else if (val instanceof ByteBufferPool) {\n      ((ByteBufferPool)val).putBuffer(buffer);\n    }\n  }\n\n  @Override\n  public synchronized void unbuffer() {\n    closeCurrentBlockReaders();\n  }\n\n  @Override\n  public boolean hasCapability(String capability) {\n    switch (StringUtils.toLowerCase(capability)) {\n    case StreamCapabilities.READAHEAD:","sourceCodeStart":1959,"sourceCodeEnd":1995,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1959-L1995","documentation":"DFSInputStream.releaseBuffer(ByteBuffer) handles buffers returned by the zero-copy read API: it removes the buffer from the stream's extendedReadBuffers map, which associates each mmap'd or pool-backed buffer with the resource that must be freed (ClientMmap or ByteBufferPool). A buffer that is not in the map - wrong stream, already released, or never produced by this stream - triggers IllegalArgumentException. The map remove() means each buffer can be released exactly once, by exactly the stream that created it.","triggerScenarios":"Calling releaseBuffer() twice with the same ByteBuffer (the first call removes it from the map, the second throws); returning a readahead buffer to a different FSDataInputStream than it was read from - typical when a cache keyed by path/block hands buffers to a re-opened stream; releasing a foreign buffer that never came from this stream.","commonSituations":"HBase-style readahead/bucket caches that pool zero-copy buffers across region/block reopens; error paths where cleanup releases a buffer that a success path already released; refactored code that moved buffer ownership between classes without moving the release.","solutions":["Release each buffer exactly once, to the stream that produced it: wrap release in a provenance-tracking map (ByteBuffer -> stream) and remove the entry as you release.","In caches, invalidate buffer entries whenever the owning stream is closed or the block re-opened, so stale buffers are never returned later.","Make the release path idempotent in your wrapper: only call stream.releaseBuffer(buf) if provenance.remove(buf) returns the owning stream.","Audit double-release paths in finally blocks: a finally that releases after the try already released is the classic source."],"exampleFix":"// before\ntry {\n  ByteBuffer b = zeroCopyIn.read(pool, len, opts);\n  consume(b);\n} finally {\n  in.releaseBuffer(bufRef); // throws if consume() already released it\n}\n\n// after\ntry {\n  ByteBuffer b = zeroCopyIn.read(pool, len, opts);\n  consume(b);\n} finally {\n  DFSInputStream owner = (DFSInputStream) bufferOwners.remove(bufRef);\n  if (owner != null) owner.releaseBuffer(bufRef); // exactly once\n}","handlingStrategy":"validation","validationCode":"Map<ByteBuffer, DFSInputStream> owners = new IdentityHashMap<>();\n\n// record provenance when the buffer is produced\nByteBuffer b = dfsIn.read(pool, len, opts);\nif (b != null) owners.put(b, dfsIn);\n\n// release only if this stream produced it, exactly once\nDFSInputStream owner = owners.remove(buffer);\nif (owner != null) {\n  owner.releaseBuffer(buffer);\n} // foreign or already-released buffers are ignored, not released","typeGuard":null,"tryCatchPattern":"try {\n  in.releaseBuffer(buf);\n} catch (IllegalArgumentException e) {\n  if (String.valueOf(e.getMessage()).contains(\"not created by this stream\")) {\n    log.warn(\"buffer double-release or foreign buffer ignored\"); // benign cleanup race\n  } else throw e;\n}","preventionTips":["Never cache zero-copy buffers across stream re-opens - release before closing the old stream.","Track buffer-to-stream provenance (IdentityHashMap) and remove entries at release time.","Make cleanup idempotent: release only what a provenance map still contains.","Do not share mmap buffers between threads that both release."],"tags":["hdfs","hdfs-client","zero-copy","buffer-management","double-release"],"backgroundTag":"buffer-double-release","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}