apache/hadoop · error · InvalidJobConfException

SequenceFileAsBinaryOutputFormat doesn't support Record Comp

Error message

SequenceFileAsBinaryOutputFormat doesn't support Record Compression

What it means

SequenceFileAsBinaryOutputFormat writes opaque byte[] keys and values, and per-record compression is not implemented for that format. checkOutputSpecs() runs during job submission and throws InvalidJobConfException when mapreduce.map.output.compress-style output compression is enabled with CompressionType.RECORD. The job fails fast at submit time rather than at task runtime.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SequenceFileAsBinaryOutputFormat.java:164

          out.appendRaw(bkey.getBytes(), 0, bkey.getLength(), wvaluebytes);
          wvaluebytes.reset(null);
        }

        public void close(Reporter reporter) throws IOException { 
          out.close();
        }

      };

  }

  @Override 
  public void checkOutputSpecs(FileSystem ignored, JobConf job) 
            throws IOException {
    super.checkOutputSpecs(ignored, job);
    if (getCompressOutput(job) && 
        getOutputCompressionType(job) == CompressionType.RECORD ){
        throw new InvalidJobConfException("SequenceFileAsBinaryOutputFormat "
                    + "doesn't support Record Compression" );
    }

  }

}

View on GitHub (pinned to 2add963021)

Solutions

Use CompressionType.BLOCK (or NONE) with SequenceFileAsBinaryOutputFormat.
If record-level compression is a hard requirement, switch to SequenceFileOutputFormat and serialize objects instead of raw bytes.
Check the compression type programmatically before submit when the format is chosen dynamically.

Example fix

// before
SequenceFileAsBinaryOutputFormat.setOutputPath(job, out);
FileOutputFormat.setCompressOutput(job, true);
SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.RECORD);

// after
SequenceFileAsBinaryOutputFormat.setOutputPath(job, out);
FileOutputFormat.setCompressOutput(job, true);
SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK);

Defensive patterns

Strategy: validation

Validate before calling

// before submit: binary output format rejects RECORD compression
if (job.getOutputFormatClass() == SequenceFileAsBinaryOutputFormat.class
    && FileOutputFormat.getCompressOutput(job)
    && SequenceFileOutputFormat.getOutputCompressionType(job) == CompressionType.RECORD) {
  throw new IllegalArgumentException(
      "SequenceFileAsBinaryOutputFormat supports only BLOCK or NONE compression");
}

Try / catch

try {
  job.submit();
} catch (InvalidJobConfException e) {
  // checkOutputSpecs failed at submit time; fix compression type to BLOCK and resubmit
}

Prevention

Default new binary-output jobs to CompressionType.BLOCK.
Do not share one compression template across output formats.
Unit-test job assembly by calling outputFormat.checkOutputSpecs() before submission in CI.

When it happens

Trigger: Job sets SequenceFileAsBinaryOutputFormat as output format, setCompressOutput(job, true) and setOutputCompressionType(job, CompressionType.RECORD) — job.submit() / JobClient.submitJob() throws.

Common situations: Compression settings copied from a SequenceFileOutputFormat job into a binary-output job; a shared job-template utility that always sets RECORD compression.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/28896ed4499939c3. Report an issue: GitHub.