{"record":{"id":"2b156fdc8dc61bf4","repo":"apache/hadoop","slug":"exception-happens-during-upload","errorCode":null,"errorMessage":"Exception happens during upload:{}","messagePattern":"Exception happens during upload:(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosOutputStream.java","lineNumber":278,"sourceCode":"    this.currBlock.moveData();\n\n    //\n    // Block this when too many active UploadPartThread\n    //\n    while ((this.blkIndex - this.eTags.size())\n        > this.uploadThreadSize) {\n      synchronized (this.blocksMap) {\n        try {\n          this.blocksMap.wait(10);\n        } catch (InterruptedException e) {\n          // ignore\n        }\n      }\n      if (this.exceptionMap.size() > 0) {\n        //\n        // Exception happens during upload\n        //\n        throw new IOException(\n            \"Exception happens during upload:\"\n                + exceptionMap);\n      }\n    }\n\n    if (this.exceptionMap.size() > 0) {\n      //\n      // Exception happens during upload\n      //\n      throw new IOException(\n          \"Exception happens during upload:\"\n              + exceptionMap);\n    }\n\n    synchronized (this.blocksMap) {\n      this.blocksMap.put(\n          this.currBlock.getBlkId(), this.currBlock);\n    }","sourceCodeStart":260,"sourceCodeEnd":296,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosOutputStream.java#L260-L296","documentation":"BosOutputStream uploads blocks asynchronously via UploadPartThread; background threads record failures into exceptionMap keyed by block. While the writer thread waits in the backpressure loop (blocksMap.wait(10)) for queue capacity, it polls exceptionMap and throws IOException 'Exception happens during upload:' + the map's toString, exposing which block failed and why.","triggerScenarios":"A part upload fails in the background (network error, 429/5xx from BOS, auth expiry) while the main thread keeps writing; the next flush/write that enters the capacity-wait loop observes the non-empty exceptionMap and fails fast.","commonSituations":"Large multipart uploads over flaky links; STS token expiring mid-upload; bucket QPS/bandwidth throttling; many concurrent uploads saturating egress so parts time out.","solutions":["Parse the exceptionMap contents embedded in the message — it maps block id to the underlying exception; fix that (credentials, throttling, network)","Retry the entire file write: the multipart session is not resumable through this stream","Enable stronger client-side retry/backoff for uploads and reduce concurrent upload threads","Use credentials that outlive the upload, or refresh tokens before starting large writes"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"static boolean isAsyncUploadFailure(IOException e) {\n  return e.getMessage() != null && e.getMessage().startsWith(\"Exception happens during upload\");\n}","tryCatchPattern":"catch (IOException e) {\n  if (isAsyncUploadFailure(e)) {\n    LOG.error(\"part failures: {}\", e.getMessage()); // message embeds block->exception map\n    fs.delete(path, false);                          // drop partial state, if any\n    rewriteFromSource(path);                         // whole-file retry\n  } else { throw e; }\n}","preventionTips":["Make writes idempotent at file level so whole-file retry is safe","Keep uploads shorter than credential TTL","Reduce upload threads when parts fail under throttling"],"tags":["bos","multipart-upload","async-upload","background-failure","hadoop"],"backgroundTag":"async-upload-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}