What causes "Exception happens during upload:{}" in apache/hadoop?

Parse the exceptionMap contents embedded in the message — it maps block id to the underlying exception; fix that (credentials, throttling, network)

apache/hadoop · error · IOException

Exception happens during upload:{}

Error message

Exception happens during upload:{}

What it means

BosOutputStream uploads blocks asynchronously via UploadPartThread; background threads record failures into exceptionMap keyed by block. While the writer thread waits in the backpressure loop (blocksMap.wait(10)) for queue capacity, it polls exceptionMap and throws IOException 'Exception happens during upload:' + the map's toString, exposing which block failed and why.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosOutputStream.java:278

    this.currBlock.moveData();

    //
    // Block this when too many active UploadPartThread
    //
    while ((this.blkIndex - this.eTags.size())
        > this.uploadThreadSize) {
      synchronized (this.blocksMap) {
        try {
          this.blocksMap.wait(10);
        } catch (InterruptedException e) {
          // ignore
        }
      }
      if (this.exceptionMap.size() > 0) {
        //
        // Exception happens during upload
        //
        throw new IOException(
            "Exception happens during upload:"
                + exceptionMap);
      }
    }

    if (this.exceptionMap.size() > 0) {
      //
      // Exception happens during upload
      //
      throw new IOException(
          "Exception happens during upload:"
              + exceptionMap);
    }

    synchronized (this.blocksMap) {
      this.blocksMap.put(
          this.currBlock.getBlkId(), this.currBlock);
    }

View on GitHub (pinned to 2add963021)

Solutions

Parse the exceptionMap contents embedded in the message — it maps block id to the underlying exception; fix that (credentials, throttling, network)
Retry the entire file write: the multipart session is not resumable through this stream
Enable stronger client-side retry/backoff for uploads and reduce concurrent upload threads
Use credentials that outlive the upload, or refresh tokens before starting large writes

Defensive patterns

Strategy: try-catch

Type guard

static boolean isAsyncUploadFailure(IOException e) {
  return e.getMessage() != null && e.getMessage().startsWith("Exception happens during upload");
}

Try / catch

catch (IOException e) {
  if (isAsyncUploadFailure(e)) {
    LOG.error("part failures: {}", e.getMessage()); // message embeds block->exception map
    fs.delete(path, false);                          // drop partial state, if any
    rewriteFromSource(path);                         // whole-file retry
  } else { throw e; }
}

Prevention

Make writes idempotent at file level so whole-file retry is safe
Keep uploads shorter than credential TTL
Reduce upload threads when parts fail under throttling

When it happens

Trigger: A part upload fails in the background (network error, 429/5xx from BOS, auth expiry) while the main thread keeps writing; the next flush/write that enters the capacity-wait loop observes the non-empty exceptionMap and fails fast.

Common situations: Large multipart uploads over flaky links; STS token expiring mid-upload; bucket QPS/bandwidth throttling; many concurrent uploads saturating egress so parts time out.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2b156fdc8dc61bf4. Report an issue: GitHub.