{"record":{"id":"1dd8ab38b3b210d2","repo":"apache/hadoop","slug":"catch-exception-when-allocating-bosblockbuffer","errorCode":null,"errorMessage":"catch exception when allocating BosBlockBuffer: ","messagePattern":"catch exception when allocating BosBlockBuffer: ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosOutputStream.java","lineNumber":147,"sourceCode":"          this.currBlock = this.blocks.take();\n          this.currBlock.setBlkId(this.blkIndex);\n          LOG.debug(\n              \"get free block: {}\", this.currBlock);\n        }\n        LOG.debug(\n            \"Block Buffer get !\"\n                + \" key is {}. blkIndex is {}.\"\n                + \" blockSize is {}\",\n            this.key, this.blkIndex,\n            this.blockSize);\n        this.bytesWrittenToBlock = 0;\n        this.blkIndex++;\n      } catch (Throwable throwable) {\n        LOG.error(\n            \"catch exception when allocating\"\n                + \" BosBlockBuffer: \",\n            throwable);\n        throw new IOException(\n            \"catch exception when allocating\"\n                + \" BosBlockBuffer: \",\n            throwable);\n      }\n    }\n  }\n\n  /**\n   * Writes a single byte to this output stream.\n   *\n   * @param b the byte to write\n   * @throws IOException if the stream is closed or an I/O\n   *                     error occurs\n   */\n  @Override\n  public synchronized void write(int b)\n      throws IOException {\n    if (this.closed) {","sourceCodeStart":129,"sourceCodeEnd":165,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosOutputStream.java#L129-L165","documentation":"BosOutputStream.createBlockBufferIfNull wraps any Throwable thrown while allocating a new BosBlockBuffer into IOException 'catch exception when allocating BosBlockBuffer' (the original is preserved as the cause and LOG.error'd). The typical cause is OutOfMemoryError from heap-allocated block buffers, but any allocator/pool failure lands here too.","triggerScenarios":"The first write/flush that needs a new block while buffer allocation throws — most commonly when blockSize × concurrent blocks/upload threads exceeds available heap (OutOfMemoryError), or a buffer-pool/factory misconfiguration throws.","commonSituations":"Large fs.bos block size combined with many upload threads per stream and many concurrent streams; NM/YARN container memory limits; jobs writing many files simultaneously from one JVM.","solutions":["Read the cause: for OutOfMemoryError reduce block size, upload-thread count, or concurrent writers; or raise container heap","Lower connector multipart settings (block size, upload thread size) so worst-case buffered bytes fit in memory","Cap concurrent BosOutputStreams (job slots, semaphore) instead of unbounded parallel file creation"],"exampleFix":"# before\nfs.bos.block.size=536870912\nfs.bos.upload.thread.size=16\n\n# after: fit worst-case = blockSize x threads x streams in heap\nfs.bos.block.size=134217728\nfs.bos.upload.thread.size=8","handlingStrategy":"try-catch","validationCode":"// worst-case buffered bytes must fit in heap before opening streams\nlong worstCase = (long) blockSize * uploadThreads * plannedStreams;\nif (worstCase > Runtime.getRuntime().maxMemory() / 2) {\n  throw new IllegalStateException(\"BOS buffer config exceeds heap: \" + worstCase);\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getCause() instanceof OutOfMemoryError) {\n    // fail the task fast;OOM means config, not transient I/O — do not retry unchanged\n  }\n  throw e;\n}","preventionTips":["Size blockSize × uploadThreads × concurrentStreams within container heap","Cap concurrent output streams per JVM","Watch for OutOfMemoryError as the cause before debugging BOS itself"],"tags":["bos","out-of-memory","buffer-allocation","output-stream","hadoop"],"backgroundTag":"resource-allocation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}