apache/beam · error · RuntimeException

Unable to setup job.

Error message

Unable to setup job.

What it means

During HadoopFormatIO write, each worker calls the OutputCommitter's setupJob (job setup). If setup fails with any exception other than FileAlreadyExistsException (which is tolerated as another worker already set it up), it is wrapped in RuntimeException('Unable to setup job.').

Solutions

1. Check the wrapped cause for the underlying Hadoop exception and fix it (permissions, path, connectivity).
2. Validate the output path is writable by the job's user before launching the pipeline.
3. Confirm HDFS/cluster connectivity and Kerberos ticket validity on workers.
4. Ensure only one job setup semantics apply — a FileAlreadyExistsException is benign, but other duplicates of setup failures indicate config issues.

Example fix

// before
// output dir pointing at non-writable location
conf.set("mapreduce.output.fileoutputformat.outputdir", "/restricted/out");
// after
Path out = new Path("/user/myuser/out");
FileSystem fs = out.getFileSystem(conf);
fs.mkdirs(out); // fails fast with clear message
conf.set("mapreduce.output.fileoutputformat.outputdir", out.toString());

Path out = new Path(conf.get("mapreduce.output.fileoutputformat.outputdir"));
FileSystem fs = out.getFileSystem(conf);
if (!fs.exists(out.getParent())) throw new IllegalStateException("parent dir missing: " + out.getParent());
fs.checkAccess(new Path(out.toUri()), FsAction.WRITE); // permission pre-check

try {
  pipeline.run().waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unable to setup job.")) {
    // check cause: FileAlreadyExistsException is benign; others need retry/fix
    if (!(e.getCause() instanceof FileAlreadyExistsException)) {
      throw new IOException("job setup failed: " + e.getCause(), e.getCause());
    }
  } else throw e;
}

When it happens

Trigger: Worker executes the write's setup step; outputCommitter.setupJob(jobContext) throws — e.g. HDFS not reachable, permissions denied creating output dir, committer-specific failures — anything that is not FileAlreadyExistsException.

Common situations: HDFS NameNode unavailable; missing write permissions on output path; invalid JobConf carried in the sink's configuration; Kerberos authentication problems; misbehaving custom OutputCommitter.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• Unable to commit job.
• Unable to create OutputCommitter object:
• Expected file path but received directory path
• Support for move options is not yet implemented.
• Unable to copy resource

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/af9c330773cef8f0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java:1783

     * @param window window
     */
    private void trySetupJob(JobID jobId, Configuration conf, BoundedWindow window) {
      try {
        TaskAttemptContext setupTaskContext = HadoopFormats.createSetupTaskContext(conf, jobId);
        OutputFormat<?, ?> jobOutputFormat = HadoopFormats.createOutputFormatFromConfig(conf);

        jobOutputFormat.checkOutputSpecs(setupTaskContext);
        jobOutputFormat.getOutputCommitter(setupTaskContext).setupJob(setupTaskContext);

        LOG.info(
            "Job with id {} successfully configured from window with max timestamp {}.",
            jobId.getJtIdentifier(),
            window.maxTimestamp());

      } catch (FileAlreadyExistsException e) {
        LOG.info("Job was already set by other worker. Skipping rest of the setup.");
      } catch (Exception e) {
        throw new RuntimeException("Unable to setup job.", e);
      }
    }
  }

  /**
   * Commits whole write job.
   *
   * @param <T> type of TaskId identifier
   */
  private static class CommitJobFn<T> extends DoFn<Iterable<T>, Void> {

    private final PCollectionView<Configuration> configView;
    private final ExternalSynchronization externalSynchronization;

    CommitJobFn(
        PCollectionView<Configuration> configView,
        ExternalSynchronization externalSynchronization) {
      this.configView = configView;

View on GitHub (pinned to 12126d8942)