{"record":{"id":"3b19c893e7fa14c8","repo":"apache/hadoop","slug":"interrupted-multi-part-upload-with-id-s-to-s","errorCode":null,"errorMessage":"Interrupted multi-part upload with id '%s' to %s","messagePattern":"Interrupted multi-part upload with id '(.+?)' to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSBlockOutputStream.java","lineNumber":751,"sourceCode":"     * Block awaiting all outstanding uploads to complete.\n     *\n     * @return list of results\n     * @throws IOException IO Problems\n     */\n    private List<PartEtag> waitForAllPartUploads() throws IOException {\n      LOG.debug(\"Waiting for {} uploads to complete\",\n          partETagsFutures.size());\n      try {\n        return Futures.allAsList(partETagsFutures).get();\n      } catch (InterruptedException ie) {\n        LOG.warn(\"Interrupted partUpload\", ie);\n        LOG.debug(\"Cancelling futures\");\n        for (ListenableFuture<PartEtag> future : partETagsFutures) {\n          future.cancel(true);\n        }\n        // abort multipartupload\n        this.abort();\n        throw new IOException(\n            \"Interrupted multi-part upload with id '\" + uploadId\n                + \"' to \" + key);\n      } catch (ExecutionException ee) {\n        // there is no way of recovering so abort\n        // cancel all partUploads\n        LOG.debug(\"While waiting for upload completion\", ee);\n        LOG.debug(\"Cancelling futures\");\n        for (ListenableFuture<PartEtag> future : partETagsFutures) {\n          future.cancel(true);\n        }\n        // abort multipartupload\n        this.abort();\n        throw OBSCommonUtils.extractException(\n            \"Multi-part upload with id '\" + uploadId + \"' to \" + key,\n            key, ee);\n      }\n    }\n","sourceCodeStart":733,"sourceCodeEnd":769,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSBlockOutputStream.java#L733-L769","documentation":"OBSBlockOutputStream's block-upload wait does Futures.allAsList(partETagsFutures).get(); an InterruptedException there means some other thread called Thread.interrupt() on the writer while it was joining part uploads. The stream then cancels all part futures (future.cancel(true)), calls this.abort() to kill the multipart upload, and throws IOException('Interrupted multi-part upload with id '<id>' to <key>'). No data is committed: the MPU is aborted and the file must be rewritten from scratch.","triggerScenarios":"Job/task cancellation (YARN preemption, Spark job cancellation) interrupting the committing thread; ExecutorService.shutdownNow() while a writer awaits part uploads; user code calling Thread.stop-style interrupts or Future.cancel(true) on the writing task; CLI tools killed with certain signals that surface as interrupts.","commonSituations":"Spark speculation killing speculative tasks mid-commit; Flink checkpoint timeouts cancelling sink writers; test harnesses that shutdownNow() executors in finally blocks; monitoring scripts killing long uploads.","solutions":["Avoid interrupting writer threads during commit: use graceful cancellation (let the current write/close finish, or cancel between files).","If interruption is legitimate (task preemption), simply accept the abort and re-run the task — the output file never existed, so the job can safely retry it idempotently.","Replace shutdownNow() with shutdown() + awaitTermination so in-flight uploads drain.","Make writes idempotent (write to temp path then atomic rename) so a retried task after cancellation does not duplicate data."],"exampleFix":"// before\nExecutorService pool = ...;\n// task uses pool.submit(writerTask) ...\npool.shutdownNow(); // interrupts writer waiting on partETagsFutures -> MPU aborted\n\n// after\npool.shutdown();\nif (!pool.awaitTermination(60, TimeUnit.SECONDS)) {\n  log.warn(\"writers did not finish; data may need re-run\");\n  pool.shutdownNow();\n}","handlingStrategy":"retry","validationCode":"// nothing to validate client-side; instead make cancellation graceful\nboolean drained = pool.awaitTerminationGracefully(); // shutdown() + awaitTermination before any cancel","typeGuard":null,"tryCatchPattern":"try {\n  out.close(); // commit waits on partETagsFutures\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).startsWith(\"Interrupted multi-part upload\")) {\n    LOG.warn(\"commit interrupted; task will be retried — output was aborted, safe to re-run\", e);\n    throw new RetryableTaskException(e); // scheduler re-executes the whole write\n  }\n  throw e;\n}","preventionTips":["Use shutdown()+awaitTermination, never shutdownNow() on pools that write.","Make writes idempotent: temp path + atomic rename on success.","Treat task preemption during commit as retryable, not as data loss — the MPU abort guarantees no partial object.","Avoid Future.cancel(true) on tasks holding output streams."],"tags":["obs","huaweicloud","interrupted","multipart-upload","concurrency"],"backgroundTag":"io-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}