apache/hadoop · error · IOException

catch exception when allocating BosBlockBuffer:

Error message

catch exception when allocating BosBlockBuffer:

What it means

BosOutputStream.createBlockBufferIfNull wraps any Throwable thrown while allocating a new BosBlockBuffer into IOException 'catch exception when allocating BosBlockBuffer' (the original is preserved as the cause and LOG.error'd). The typical cause is OutOfMemoryError from heap-allocated block buffers, but any allocator/pool failure lands here too.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosOutputStream.java:147

          this.currBlock = this.blocks.take();
          this.currBlock.setBlkId(this.blkIndex);
          LOG.debug(
              "get free block: {}", this.currBlock);
        }
        LOG.debug(
            "Block Buffer get !"
                + " key is {}. blkIndex is {}."
                + " blockSize is {}",
            this.key, this.blkIndex,
            this.blockSize);
        this.bytesWrittenToBlock = 0;
        this.blkIndex++;
      } catch (Throwable throwable) {
        LOG.error(
            "catch exception when allocating"
                + " BosBlockBuffer: ",
            throwable);
        throw new IOException(
            "catch exception when allocating"
                + " BosBlockBuffer: ",
            throwable);
      }
    }
  }

  /**
   * Writes a single byte to this output stream.
   *
   * @param b the byte to write
   * @throws IOException if the stream is closed or an I/O
   *                     error occurs
   */
  @Override
  public synchronized void write(int b)
      throws IOException {
    if (this.closed) {

View on GitHub (pinned to 2add963021)

Solutions

  1. Read the cause: for OutOfMemoryError reduce block size, upload-thread count, or concurrent writers; or raise container heap
  2. Lower connector multipart settings (block size, upload thread size) so worst-case buffered bytes fit in memory
  3. Cap concurrent BosOutputStreams (job slots, semaphore) instead of unbounded parallel file creation

Example fix

# before
fs.bos.block.size=536870912
fs.bos.upload.thread.size=16

# after: fit worst-case = blockSize x threads x streams in heap
fs.bos.block.size=134217728
fs.bos.upload.thread.size=8
Defensive patterns

Strategy: try-catch

Validate before calling

// worst-case buffered bytes must fit in heap before opening streams
long worstCase = (long) blockSize * uploadThreads * plannedStreams;
if (worstCase > Runtime.getRuntime().maxMemory() / 2) {
  throw new IllegalStateException("BOS buffer config exceeds heap: " + worstCase);
}

Try / catch

catch (IOException e) {
  if (e.getCause() instanceof OutOfMemoryError) {
    // fail the task fast;OOM means config, not transient I/O — do not retry unchanged
  }
  throw e;
}

Prevention

When it happens

Trigger: The first write/flush that needs a new block while buffer allocation throws — most commonly when blockSize × concurrent blocks/upload threads exceeds available heap (OutOfMemoryError), or a buffer-pool/factory misconfiguration throws.

Common situations: Large fs.bos block size combined with many upload threads per stream and many concurrent streams; NM/YARN container memory limits; jobs writing many files simultaneously from one JVM.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1dd8ab38b3b210d2. Report an issue: GitHub.