{"record":{"id":"837902cd89ec3d49","repo":"apache/hadoop","slug":"sequencefileasbinaryoutputformat-doesn-t-support-r-837902","errorCode":null,"errorMessage":"SequenceFileAsBinaryOutputFormat doesn't support Record Compression","messagePattern":"SequenceFileAsBinaryOutputFormat doesn't support Record Compression","errorType":"validation","errorClass":"InvalidJobConfException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/SequenceFileAsBinaryOutputFormat.java","lineNumber":161,"sourceCode":"      public void write(BytesWritable bkey, BytesWritable bvalue)\n        throws IOException {\n        wvaluebytes.reset(bvalue);\n        out.appendRaw(bkey.getBytes(), 0, bkey.getLength(), wvaluebytes);\n        wvaluebytes.reset(null);\n      }\n\n      public void close(TaskAttemptContext context) throws IOException { \n        out.close();\n      }\n    };\n  }\n\n  @Override \n  public void checkOutputSpecs(JobContext job) throws IOException {\n    super.checkOutputSpecs(job);\n    if (getCompressOutput(job) && \n        getOutputCompressionType(job) == CompressionType.RECORD ) {\n      throw new InvalidJobConfException(\"SequenceFileAsBinaryOutputFormat \"\n        + \"doesn't support Record Compression\" );\n    }\n  }\n}\n","sourceCodeStart":143,"sourceCodeEnd":166,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/SequenceFileAsBinaryOutputFormat.java#L143-L166","documentation":"SequenceFileAsBinaryOutputFormat.checkOutputSpecs() validates the job configuration at submission (and again task-side) and throws InvalidJobConfException when output compression is on AND the SequenceFile compression type is RECORD, because the binary adapter only supports NONE and BLOCK. This is a deliberate fail-fast: without it the task would die mid-write with the UnsupportedOperationException from error 4344.","triggerScenarios":"FileOutputFormat.getCompressOutput(job)==true and getOutputCompressionType(job)==CompressionType.RECORD with SequenceFileAsBinaryOutputFormat as the output format. Note RECORD is the SequenceFile default, so merely enabling compression without choosing a type triggers it.","commonSituations":"Reusing a job template built for SequenceFileOutputFormat with record compression; enabling mapreduce.output.fileoutputformat.compress=true cluster-wide and hitting jobs that write binary key/values; forgetting that compression type must be set explicitly to BLOCK.","solutions":["Set the compression type to BLOCK: SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK).","Or turn compression off for this job: FileOutputFormat.setCompressOutput(job, false).","Run job submission through the normal path (job.waitForCompletion) so checkOutputSpecs rejects the config locally before consuming cluster slots.","Audit shared conf templates for compression.type=RECORD defaults."],"exampleFix":"// before: compression on, type defaults to RECORD\nFileOutputFormat.setCompressOutput(job, true);\njob.setOutputFormatClass(SequenceFileAsBinaryOutputFormat.class);\n\n// after: pick a supported compression type\nFileOutputFormat.setCompressOutput(job, true);\nSequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK);\njob.setOutputFormatClass(SequenceFileAsBinaryOutputFormat.class);","handlingStrategy":"validation","validationCode":"// client-side pre-flight mirroring checkOutputSpecs\nJob j = Job.getInstance(conf);\nif (j.getOutputFormatClass() == SequenceFileAsBinaryOutputFormat.class\n    && FileOutputFormat.getCompressOutput(j)\n    && FileOutputFormat.getOutputCompressionType(j) == CompressionType.RECORD) {\n  throw new IllegalArgumentException(\n      \"Use CompressionType.BLOCK (or disable compression) for SequenceFileAsBinaryOutputFormat\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set compression type explicitly in code rather than relying on cluster defaults.","Lint shared job templates for compression.type=RECORD.","Expect this at job.waitForCompletion (submission validates specs) - fix config, don't catch-and-continue."],"tags":["hadoop","sequencefile","compression","mapreduce","job-configuration","fail-fast"],"backgroundTag":"unsupported-compression-type","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}