apache/hadoop · error · IOException

Output Format not set for LazyOutputFormat

Error message

Output Format not set for LazyOutputFormat

What it means

Thrown as IOException from LazyOutputFormat.getBaseOutputFormat (LazyOutputFormat.java:64). LazyOutputFormat only creates output files when the first record is written; it needs the real OutputFormat class in mapreduce.output.lazyoutputformat.outputformat. If that key is absent, ReflectionUtils.newInstance(null, conf) yields null and this IOException is raised when getRecordWriter/checkOutputSpecs/getOutputCommitter runs.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/LazyOutputFormat.java:64

   * Set the underlying output format for LazyOutputFormat.
   * @param job the {@link Job} to modify
   * @param theClass the underlying class
   */
  @SuppressWarnings("unchecked")
  public static void  setOutputFormatClass(Job job, 
                                     Class<? extends OutputFormat> theClass) {
      job.setOutputFormatClass(LazyOutputFormat.class);
      job.getConfiguration().setClass(OUTPUT_FORMAT, 
          theClass, OutputFormat.class);
  }

  @SuppressWarnings("unchecked")
  private void getBaseOutputFormat(Configuration conf) 
  throws IOException {
    baseOut =  ((OutputFormat<K, V>) ReflectionUtils.newInstance(
      conf.getClass(OUTPUT_FORMAT, null), conf));
    if (baseOut == null) {
      throw new IOException("Output Format not set for LazyOutputFormat");
    }
  }

  @Override
  public RecordWriter<K, V> getRecordWriter(TaskAttemptContext context)
  throws IOException, InterruptedException {
    if (baseOut == null) {
      getBaseOutputFormat(context.getConfiguration());
    }
    return new LazyRecordWriter<K, V>(baseOut, context);
  }
  
  @Override
  public void checkOutputSpecs(JobContext context) 
  throws IOException, InterruptedException {
    if (baseOut == null) {
      getBaseOutputFormat(context.getConfiguration());
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Configure via the API: LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class) — never set the LazyOutputFormat class by hand
2. If configuring raw properties, set both: mapreduce.job.outputformat.class=org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat AND mapreduce.output.lazyoutputformat.outputformat=<delegate FQCN>
3. Check for config whitelisting/scrubbing in your framework that strips the lazyoutputformat key before tasks launch

Example fix

// before: wrapper registered without its delegate
job.setOutputFormatClass(LazyOutputFormat.class);

// after: helper sets wrapper + delegate key together
LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class);

// before submit: both keys must be present
Configuration conf = job.getConfiguration();
if (job.getOutputFormatClass().equals(LazyOutputFormat.class)
    && conf.get("mapreduce.output.lazyoutputformat.outputformat") == null) {
  throw new IllegalStateException("LazyOutputFormat configured without its delegate");
}

When it happens

Trigger: Setting job.setOutputFormatClass(LazyOutputFormat.class) manually (or via conf) instead of calling LazyOutputFormat.setOutputFormatClass(job, ActualFormat.class); the latter both registers LazyOutputFormat AND stores the delegate under OUTPUT_FORMAT. Also losing the key when a Configuration is rebuilt/filtered (e.g. whitelisting config keys for tasks) before task start.

Common situations: Recipes copied from docs that show setOutputFormatClass(LazyOutputFormat.class) plus a hand-written conf key with a typo; security-filtered configurations (Configuration.loadResources / mapred-site trimming) dropping the key; using LazyOutputFormat together with MultipleOutputs but wiring it manually.

Related errors

• Invalid specification for distributed-cache artifacts of typ
• Unable to parse '{}' as a URI, check the setting for mapredu
• Could not locate MapReduce framework name '{}' in mapreduce.
• MapReduce JobHistory WebApp Address does not contain a valid
• Type mismatch in key from map: expected {keyClassName}, rece

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/113bc14c871f5a06. Report an issue: GitHub.