{"record":{"id":"de463a77c717e8fa","repo":"apache/hadoop","slug":"zero-copy-reads-were-not-available-and-the-bytebu","errorCode":null,"errorMessage":"zero-copy reads were not available, and the ByteBufferPool did not provide us with {} buffer.","messagePattern":"zero-copy reads were not available, and the ByteBufferPool did not provide us with (.+?) buffer\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferUtil.java","lineNumber":68,"sourceCode":"   *\n   * @param stream input stream.\n   * @param bufferPool bufferPool.\n   * @param maxLength maxLength.\n   * @throws IOException raised on errors performing I/O.\n   * @return byte buffer.\n   */\n  public static ByteBuffer fallbackRead(\n      InputStream stream, ByteBufferPool bufferPool, int maxLength)\n          throws IOException {\n    if (bufferPool == null) {\n      throw new UnsupportedOperationException(\"zero-copy reads \" +\n          \"were not available, and you did not provide a fallback \" +\n          \"ByteBufferPool.\");\n    }\n    boolean useDirect = streamHasByteBufferRead(stream);\n    ByteBuffer buffer = bufferPool.getBuffer(useDirect, maxLength);\n    if (buffer == null) {\n      throw new UnsupportedOperationException(\"zero-copy reads \" +\n          \"were not available, and the ByteBufferPool did not provide \" +\n          \"us with \" + (useDirect ? \"a direct\" : \"an indirect\") +\n          \"buffer.\");\n    }\n    Preconditions.checkState(buffer.capacity() > 0);\n    Preconditions.checkState(buffer.isDirect() == useDirect);\n    maxLength = Math.min(maxLength, buffer.capacity());\n    boolean success = false;\n    try {\n      if (useDirect) {\n        buffer.clear();\n        buffer.limit(maxLength);\n        ByteBufferReadable readable = (ByteBufferReadable)stream;\n        int totalRead = 0;\n        while (true) {\n          if (totalRead >= maxLength) {\n            success = true;\n            break;","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferUtil.java#L50-L86","documentation":"After fallbackRead obtains a pool, it calls bufferPool.getBuffer(useDirect, maxLength); the pool's contract permits returning null, but the fallback read cannot proceed without a buffer, so it throws UnsupportedOperationException stating that the pool did not provide a direct or indirect buffer. `useDirect` is true when the stream implements ByteBufferReadable - the pool must be able to supply that flavor.","triggerScenarios":"read(pool, maxLength) on a non-zero-copy stream with a bounded pool that returns null when exhausted or refuses direct buffers; a custom pool whose getBuffer returns null for the requested flavor; pools starved because buffers are never released back via releaseBuffer.","commonSituations":"Hand-rolled pools tuned for indirect buffers meet a ByteBufferReadable stream that demands direct ones; long-lived readers leak pooled buffers (missing releaseBuffer) until the pool runs dry; tests with zero-capacity pools.","solutions":["Use ElasticByteBufferPool, which never returns null and allocates buffers as needed.","Fix the custom pool to allocate (or block) instead of returning null in the fallback read path, and honor the useDirect flavor request.","Ensure every buffer handed out is returned with releaseBuffer once consumed (including on exception paths).","Fall back to byte[] reads when the pool cannot serve the request."],"exampleFix":"// before\nByteBuffer b = in.read(boundedPool, 65536); // pool returns null -> UOE\n\n// after\nByteBuffer b = in.read(new ElasticByteBufferPool(), 65536);","handlingStrategy":"fallback","validationCode":"static ByteBuffer getBufferOrFail(ByteBufferPool pool, boolean direct, int len) {\n  ByteBuffer b = pool.getBuffer(direct, len);\n  if (b == null) {\n    b = direct ? ByteBuffer.allocateDirect(len) : ByteBuffer.allocate(len);\n  }\n  return b;\n}","typeGuard":null,"tryCatchPattern":"try {\n  buf = in.read(pool, maxLength);\n} catch (UnsupportedOperationException e) {\n  buf = in.read(new org.apache.hadoop.io.ElasticByteBufferPool(), maxLength);\n}","preventionTips":["Prefer ElasticByteBufferPool, which never returns null","Return every borrowed buffer via releaseBuffer, including on exception paths","If you write a custom pool, honor the useDirect flavor request and allocate instead of returning null on the read path"],"tags":["hadoop","filesystem","zero-copy","bytebuffer","memory-pool"],"backgroundTag":"bytebuffer-pool-exhausted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}