apache/hadoop · error · UnsupportedOperationException

WritableValueBytes doesn't support RECORD compression

Error message

WritableValueBytes doesn't support RECORD compression

What it means

SequenceFileAsBinaryOutputFormat writes raw BytesWritable keys/values through an adapter (WritableValueBytes). SequenceFile record compression needs per-value compressed writes, which the adapter cannot do, so writeCompressedBytes() throws UnsupportedOperationException. Normally this configuration is rejected up front by checkOutputSpecs (InvalidJobConfException, error 4345); this runtime exception only appears when a writer is driven with RECORD compression through a path that bypassed that check (custom record-writer wiring, tests, direct SequenceFile.Writer-style usage).

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/SequenceFileAsBinaryOutputFormat.java:73

      this.value = null;
    }
    
    public WritableValueBytes(BytesWritable value) {
      this.value = value;
    }

    public void reset(BytesWritable value) {
      this.value = value;
    }

    public void writeUncompressedBytes(DataOutputStream outStream)
        throws IOException {
      outStream.write(value.getBytes(), 0, value.getLength());
    }

    public void writeCompressedBytes(DataOutputStream outStream)
        throws IllegalArgumentException, IOException {
      throw new UnsupportedOperationException(
        "WritableValueBytes doesn't support RECORD compression"); 
    }
    
    public int getSize(){
      return value.getLength();
    }
  }

  /**
   * Set the key class for the {@link SequenceFile}
   * <p>This allows the user to specify the key class to be different 
   * from the actual class ({@link BytesWritable}) used for writing </p>
   * 
   * @param job the {@link Job} to modify
   * @param theClass the SequenceFile output key class.
   */
  static public void setSequenceFileOutputKeyClass(Job job, 
      Class<?> theClass) {

View on GitHub (pinned to 2add963021)

Solutions

Switch compression type to BLOCK: conf.set("mapreduce.output.fileoutputformat.compression.type", "BLOCK").
Or disable compression for this output: FileOutputFormat.setCompressOutput(job, false).
Let the framework call checkOutputSpecs (it fails fast at submit) instead of bypassing job submission.

Example fix

// before
conf.setBoolean("mapreduce.output.fileoutputformat.compress", true);
conf.set("mapreduce.output.fileoutputformat.compression.type", "RECORD");

// after
conf.setBoolean("mapreduce.output.fileoutputformat.compress", true);
conf.set("mapreduce.output.fileoutputformat.compression.type", "BLOCK");

Defensive patterns

Strategy: validation

Validate before calling

// guard before obtaining/using the record writer
if (FileOutputFormat.getCompressOutput(job)
    && FileOutputFormat.getOutputCompressionType(job) == CompressionType.RECORD) {
  throw new InvalidJobConfException(
      "SequenceFileAsBinaryOutputFormat supports only NONE or BLOCK compression");
}

Prevention

Always pair setCompressOutput(true) with an explicit setOutputCompressionType(BLOCK) for binary sequence output.
Prefer the framework's checkOutputSpecs path so misconfiguration fails at submit, not at write time.
Remember RECORD is the SequenceFile default type - enabling compression alone is not safe.

When it happens

Trigger: The record writer reaching writeCompressedBytes() because output compression is enabled with CompressionType.RECORD - i.e. mapreduce.output.fileoutputformat.compress=true and mapreduce.output.fileoutputformat.compression.type=RECORD while writing via SequenceFileAsBinaryOutputFormat's BinarySequenceFileRecordWriter outside the normal checked submit path.

Common situations: Test code constructing the RecordWriter directly; a job where compression settings were changed after checkOutputSpecs ran; shared job-conf templates that default compression type to RECORD (the SequenceFile default).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9277dcae17c418a3. Report an issue: GitHub.