apache/hadoop · error · InvalidJobConfException

SequenceFileAsBinaryOutputFormat doesn't support Record Comp

Error message

SequenceFileAsBinaryOutputFormat doesn't support Record Compression

What it means

SequenceFileAsBinaryOutputFormat.checkOutputSpecs() validates the job configuration at submission (and again task-side) and throws InvalidJobConfException when output compression is on AND the SequenceFile compression type is RECORD, because the binary adapter only supports NONE and BLOCK. This is a deliberate fail-fast: without it the task would die mid-write with the UnsupportedOperationException from error 4344.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/SequenceFileAsBinaryOutputFormat.java:161

      public void write(BytesWritable bkey, BytesWritable bvalue)
        throws IOException {
        wvaluebytes.reset(bvalue);
        out.appendRaw(bkey.getBytes(), 0, bkey.getLength(), wvaluebytes);
        wvaluebytes.reset(null);
      }

      public void close(TaskAttemptContext context) throws IOException { 
        out.close();
      }
    };
  }

  @Override 
  public void checkOutputSpecs(JobContext job) throws IOException {
    super.checkOutputSpecs(job);
    if (getCompressOutput(job) && 
        getOutputCompressionType(job) == CompressionType.RECORD ) {
      throw new InvalidJobConfException("SequenceFileAsBinaryOutputFormat "
        + "doesn't support Record Compression" );
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Set the compression type to BLOCK: SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK).
2. Or turn compression off for this job: FileOutputFormat.setCompressOutput(job, false).
3. Run job submission through the normal path (job.waitForCompletion) so checkOutputSpecs rejects the config locally before consuming cluster slots.
4. Audit shared conf templates for compression.type=RECORD defaults.

Example fix

// before: compression on, type defaults to RECORD
FileOutputFormat.setCompressOutput(job, true);
job.setOutputFormatClass(SequenceFileAsBinaryOutputFormat.class);

// after: pick a supported compression type
FileOutputFormat.setCompressOutput(job, true);
SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK);
job.setOutputFormatClass(SequenceFileAsBinaryOutputFormat.class);

// client-side pre-flight mirroring checkOutputSpecs
Job j = Job.getInstance(conf);
if (j.getOutputFormatClass() == SequenceFileAsBinaryOutputFormat.class
    && FileOutputFormat.getCompressOutput(j)
    && FileOutputFormat.getOutputCompressionType(j) == CompressionType.RECORD) {
  throw new IllegalArgumentException(
      "Use CompressionType.BLOCK (or disable compression) for SequenceFileAsBinaryOutputFormat");
}

When it happens

Trigger: FileOutputFormat.getCompressOutput(job)==true and getOutputCompressionType(job)==CompressionType.RECORD with SequenceFileAsBinaryOutputFormat as the output format. Note RECORD is the SequenceFile default, so merely enabling compression without choosing a type triggers it.

Common situations: Reusing a job template built for SequenceFileOutputFormat with record compression; enabling mapreduce.output.fileoutputformat.compress=true cluster-wide and hitting jobs that write binary key/values; forgetting that compression type must be set explicitly to BLOCK.

Related errors

• SequenceFileAsBinaryOutputFormat doesn't support Record Comp
• WritableValueBytes doesn't support RECORD compression
• All merged files must be compressed or not.
• Compression codec {} was not found.
• Output directory not set.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/837902cd89ec3d49. Report an issue: GitHub.