{"record":{"id":"50332143dc43d723","repo":"apache/beam","slug":"unable-to-create-recordwriter-object","errorCode":null,"errorMessage":"Unable to create RecordWriter object: ","messagePattern":"Unable to create RecordWriter object: ","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java","lineNumber":1572,"sourceCode":"      try {\n        outputCommitter.abortTask(taskAttemptContext);\n      } catch (IOException e) {\n        throw new IllegalStateException(\n            String.format(\"Unable to abort task %s of job %s\", getTaskId(), getJobId()));\n      }\n    }\n\n    private RecordWriter<KeyT, ValueT> initRecordWriter(\n        OutputFormat<KeyT, ValueT> outputFormatObj, TaskAttemptContext taskAttemptContext)\n        throws IllegalStateException {\n      try {\n        LOG.info(\n            \"Creating new RecordWriter for task {} of Job with id {}.\",\n            taskAttemptContext.getTaskAttemptID().getTaskID().getId(),\n            taskAttemptContext.getJobID().getJtIdentifier());\n        return outputFormatObj.getRecordWriter(taskAttemptContext);\n      } catch (InterruptedException | IOException e) {\n        throw new IllegalStateException(\"Unable to create RecordWriter object: \", e);\n      }\n    }\n\n    private static OutputCommitter initOutputCommitter(\n        OutputFormat<?, ?> outputFormatObj,\n        Configuration conf,\n        TaskAttemptContext taskAttemptContext)\n        throws IllegalStateException {\n      OutputCommitter outputCommitter;\n      try {\n        outputCommitter = outputFormatObj.getOutputCommitter(taskAttemptContext);\n        if (outputCommitter != null) {\n          outputCommitter.setupJob(new JobContextImpl(conf, taskAttemptContext.getJobID()));\n        }\n      } catch (Exception e) {\n        throw new IllegalStateException(\"Unable to create OutputCommitter object: \", e);\n      }\n","sourceCodeStart":1554,"sourceCodeEnd":1590,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java#L1554-L1590","documentation":"The sink initializes a RecordWriter via OutputFormat.getRecordWriter(taskAttemptContext). If that call throws an IOException or InterruptedException, this IllegalStateException is thrown, wrapping the cause.","triggerScenarios":"OutputFormat.getRecordWriter() fails due to unwritable output path, existing output directory (with FileOutputFormat's fail-if-exists), missing compression codec, or interruption while creating the writer.","commonSituations":"Output directory already exists; missing HDFS write permissions; misconfigured output format/compression settings; Hadoop version mismatch in the OutputFormat class.","solutions":["Ensure the output path does not already exist (or configure a committer/format that overwrites)","Verify write permissions on the output filesystem","Check the chained cause for the concrete IOException (codec, path, config) and fix the job configuration","Validate the configured OutputFormat class and its compression settings"],"exampleFix":"// before\np.apply(\"Write\", HadoopIO.write().to(\"hdfs://out/result\")); // out/result already exists\n// after\ndeleteExistingOutputDir(\"hdfs://out/result\"); // or write to a new unique path\np.apply(\"Write\", HadoopIO.write().to(\"hdfs://out/result-\" + runId));","handlingStrategy":"validation","validationCode":"Path out = new Path(outputDir);\nFileSystem fs = out.getFileSystem(conf);\nif (fs.exists(out)) throw new IllegalStateException(\"Output dir exists: \" + out);\nif (!fs.mkdirs(out.getParent())) throw new IllegalStateException(\"Cannot create parent: \" + out.getParent());","typeGuard":null,"tryCatchPattern":"try {\n  pipeline.run().waitUntilFinish();\n} catch (Exception e) {\n  if (hasCauseMessage(e, \"Unable to create RecordWriter object\")) {\n    throw new IllegalStateException(\"Check output path, permissions, and output format config\", e);\n  }\n  throw e;\n}","preventionTips":["Delete or version the output directory before each run","Grant the job user write permissions on the output path","Pre-validate compression codec and OutputFormat configuration in a dry run"],"tags":["hadoop","recordwriter","output","sink"],"backgroundTag":"file-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}