What causes "Task set failed with an uncaught throwable" in apache/hadoop?

Search the task and worker logs around this exception for the true cause: OOM messages, 'Failed to clean up on failure', 'Failed to revert task', or linkage-error stack traces — the RuntimeException itself carries no cause

apache/hadoop · error · RuntimeException

Task set failed with an uncaught throwable

Error message

Task set failed with an uncaught throwable

What it means

Tasks (a retry/parallel helper copied from Apache Iceberg and used by the TOS committer for commit/cleanup) throws this generic RuntimeException from its single-threaded run path when the failure flag indicates a throwable escaped, but no Exception was captured to rethrow. Per the code comment, `threw` stays true only for throwables NOT caught by the per-item catch block, i.e. an Error (OOM, linkage errors) or a failure of the items iterable itself. The real cause is only visible in surrounding logs.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/common/Tasks.java:272

              try {
                abortTask.run(iterator.next());
              } catch (Exception e) {
                failed = true;
                LOG.error("Failed to abort task", e);
                // keep going
              }
              if (stopAbortsOnFailure && failed) {
                break;
              }
            }
          }
        }
      }

      if (throwFailureWhenFinished && !exceptions.isEmpty()) {
        Tasks.throwOne(exceptions, exceptionClass);
      } else if (throwFailureWhenFinished && threw) {
        throw new RuntimeException("Task set failed with an uncaught throwable");
      }

      return !threw;
    }

    private void tryRunOnFailure(I item, Exception failure) {
      try {
        onFailure.run(item, failure);
      } catch (Exception failException) {
        failure.addSuppressed(failException);
        LOG.error("Failed to clean up on failure", failException);
        // keep going
      }
    }

    private <E extends Exception> boolean runParallel(
        final Task<I, E> task, Class<E> exceptionClass) throws E {
      final Queue<I> succeeded = new ConcurrentLinkedQueue<>();

View on GitHub (pinned to 2add963021)

Solutions

Search the task and worker logs around this exception for the true cause: OOM messages, 'Failed to clean up on failure', 'Failed to revert task', or linkage-error stack traces — the RuntimeException itself carries no cause
If the underlying throwable is OutOfMemoryError, raise the task/container heap (mapreduce.map/reduce.memory.mb and java.opts) so it never occurs
For NoSuchMethodError/NoClassDefFoundError, align jar versions: put a single matching version of the TOS SDK and hadoop-tos on the classpath and remove duplicates
Configure real retries on the builder (.retry(n).exponentialBackoff(...)) so transient failures flow through the typed-exception path and are rethrown with their actual cause

Defensive patterns

Strategy: try-catch

Try / catch

try {
  new Tasks.Builder<>(items)
      .retry(3).exponentialBackoff(1000, 600000, 2.0)
      .run(task);
} catch (RuntimeException e) {
  if ("Task set failed with an uncaught throwable".equals(e.getMessage())) {
    // the original Error was never captured: go find it in the worker logs (OOM, linkage errors)
    LOG.error("Uncaught Error inside task set; check heap/classpath. Items: {}", items, e);
  }
  throw e;
}

Prevention

Wrap task bodies so Throwables are captured as Exceptions: try { ... } catch (Throwable t) { throw new RuntimeException(t); }
Size task/mapreduce heap generously when tasks buffer data — OOM is the most common uncaught Error here
Pin exact versions of the TOS SDK and hadoop-tos jars to avoid NoSuchMethodError inside tasks
Make the items iterable cheap and safe: build the list before entering the builder so iterator failures cannot occur

When it happens

Trigger: new Tasks.Builder(items)...run(task) without an executor, where a task body throws an Error/Throwable instead of an Exception (Out OfMemoryError, NoSuchMethodError, StackOverflowError), or items.iterator() throws while iterating; throwFailureWhenFinished is enabled (default true) so the failure is surfaced at the end.

Common situations: Abort/cleanup phases of the TOS committer running with an undersized task heap (OOM inside an abort-upload task); classpath conflicts between the TOS SDK and Hadoop producing NoSuchMethodError/NoClassDefFoundError inside a task lambda; lazily-built item collections that throw during iteration.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8ced6d835bf187b0. Report an issue: GitHub.