{"record":{"id":"af9c330773cef8f0","repo":"apache/beam","slug":"unable-to-setup-job","errorCode":null,"errorMessage":"Unable to setup job.","messagePattern":"Unable to setup job\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java","lineNumber":1783,"sourceCode":"     * @param window window\n     */\n    private void trySetupJob(JobID jobId, Configuration conf, BoundedWindow window) {\n      try {\n        TaskAttemptContext setupTaskContext = HadoopFormats.createSetupTaskContext(conf, jobId);\n        OutputFormat<?, ?> jobOutputFormat = HadoopFormats.createOutputFormatFromConfig(conf);\n\n        jobOutputFormat.checkOutputSpecs(setupTaskContext);\n        jobOutputFormat.getOutputCommitter(setupTaskContext).setupJob(setupTaskContext);\n\n        LOG.info(\n            \"Job with id {} successfully configured from window with max timestamp {}.\",\n            jobId.getJtIdentifier(),\n            window.maxTimestamp());\n\n      } catch (FileAlreadyExistsException e) {\n        LOG.info(\"Job was already set by other worker. Skipping rest of the setup.\");\n      } catch (Exception e) {\n        throw new RuntimeException(\"Unable to setup job.\", e);\n      }\n    }\n  }\n\n  /**\n   * Commits whole write job.\n   *\n   * @param <T> type of TaskId identifier\n   */\n  private static class CommitJobFn<T> extends DoFn<Iterable<T>, Void> {\n\n    private final PCollectionView<Configuration> configView;\n    private final ExternalSynchronization externalSynchronization;\n\n    CommitJobFn(\n        PCollectionView<Configuration> configView,\n        ExternalSynchronization externalSynchronization) {\n      this.configView = configView;","sourceCodeStart":1765,"sourceCodeEnd":1801,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java#L1765-L1801","documentation":"During HadoopFormatIO write, each worker calls the OutputCommitter's setupJob (job setup). If setup fails with any exception other than FileAlreadyExistsException (which is tolerated as another worker already set it up), it is wrapped in RuntimeException('Unable to setup job.').","triggerScenarios":"Worker executes the write's setup step; outputCommitter.setupJob(jobContext) throws — e.g. HDFS not reachable, permissions denied creating output dir, committer-specific failures — anything that is not FileAlreadyExistsException.","commonSituations":"HDFS NameNode unavailable; missing write permissions on output path; invalid JobConf carried in the sink's configuration; Kerberos authentication problems; misbehaving custom OutputCommitter.","solutions":["Check the wrapped cause for the underlying Hadoop exception and fix it (permissions, path, connectivity).","Validate the output path is writable by the job's user before launching the pipeline.","Confirm HDFS/cluster connectivity and Kerberos ticket validity on workers.","Ensure only one job setup semantics apply — a FileAlreadyExistsException is benign, but other duplicates of setup failures indicate config issues."],"exampleFix":"// before\n// output dir pointing at non-writable location\nconf.set(\"mapreduce.output.fileoutputformat.outputdir\", \"/restricted/out\");\n// after\nPath out = new Path(\"/user/myuser/out\");\nFileSystem fs = out.getFileSystem(conf);\nfs.mkdirs(out); // fails fast with clear message\nconf.set(\"mapreduce.output.fileoutputformat.outputdir\", out.toString());","handlingStrategy":"retry","validationCode":"Path out = new Path(conf.get(\"mapreduce.output.fileoutputformat.outputdir\"));\nFileSystem fs = out.getFileSystem(conf);\nif (!fs.exists(out.getParent())) throw new IllegalStateException(\"parent dir missing: \" + out.getParent());\nfs.checkAccess(new Path(out.toUri()), FsAction.WRITE); // permission pre-check","typeGuard":null,"tryCatchPattern":"try {\n  pipeline.run().waitUntilFinish();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Unable to setup job.\")) {\n    // check cause: FileAlreadyExistsException is benign; others need retry/fix\n    if (!(e.getCause() instanceof FileAlreadyExistsException)) {\n      throw new IOException(\"job setup failed: \" + e.getCause(), e.getCause());\n    }\n  } else throw e;\n}","preventionTips":["Verify HDFS connectivity and Kerberos credentials on all workers before the run.","Use unique per-run output paths (UUID suffix) to avoid setup conflicts.","Confirm the job user has write permission on the output directory.","Keep the embedded JobConf minimal and validated (test with hadoop CLI first)."],"tags":["hadoop","hadoopformatio","job-setup","hdfs"],"backgroundTag":"invalid-config-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}