{"record":{"id":"28896ed4499939c3","repo":"apache/hadoop","slug":"sequencefileasbinaryoutputformat-doesn-t-support-r","errorCode":null,"errorMessage":"SequenceFileAsBinaryOutputFormat doesn't support Record Compression","messagePattern":"SequenceFileAsBinaryOutputFormat doesn't support Record Compression","errorType":"validation","errorClass":"InvalidJobConfException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SequenceFileAsBinaryOutputFormat.java","lineNumber":164,"sourceCode":"          out.appendRaw(bkey.getBytes(), 0, bkey.getLength(), wvaluebytes);\n          wvaluebytes.reset(null);\n        }\n\n        public void close(Reporter reporter) throws IOException { \n          out.close();\n        }\n\n      };\n\n  }\n\n  @Override \n  public void checkOutputSpecs(FileSystem ignored, JobConf job) \n            throws IOException {\n    super.checkOutputSpecs(ignored, job);\n    if (getCompressOutput(job) && \n        getOutputCompressionType(job) == CompressionType.RECORD ){\n        throw new InvalidJobConfException(\"SequenceFileAsBinaryOutputFormat \"\n                    + \"doesn't support Record Compression\" );\n    }\n\n  }\n\n}\n","sourceCodeStart":146,"sourceCodeEnd":171,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SequenceFileAsBinaryOutputFormat.java#L146-L171","documentation":"SequenceFileAsBinaryOutputFormat writes opaque byte[] keys and values, and per-record compression is not implemented for that format. checkOutputSpecs() runs during job submission and throws InvalidJobConfException when mapreduce.map.output.compress-style output compression is enabled with CompressionType.RECORD. The job fails fast at submit time rather than at task runtime.","triggerScenarios":"Job sets SequenceFileAsBinaryOutputFormat as output format, setCompressOutput(job, true) and setOutputCompressionType(job, CompressionType.RECORD) — job.submit() / JobClient.submitJob() throws.","commonSituations":"Compression settings copied from a SequenceFileOutputFormat job into a binary-output job; a shared job-template utility that always sets RECORD compression.","solutions":["Use CompressionType.BLOCK (or NONE) with SequenceFileAsBinaryOutputFormat.","If record-level compression is a hard requirement, switch to SequenceFileOutputFormat and serialize objects instead of raw bytes.","Check the compression type programmatically before submit when the format is chosen dynamically."],"exampleFix":"// before\nSequenceFileAsBinaryOutputFormat.setOutputPath(job, out);\nFileOutputFormat.setCompressOutput(job, true);\nSequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.RECORD);\n\n// after\nSequenceFileAsBinaryOutputFormat.setOutputPath(job, out);\nFileOutputFormat.setCompressOutput(job, true);\nSequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK);","handlingStrategy":"validation","validationCode":"// before submit: binary output format rejects RECORD compression\nif (job.getOutputFormatClass() == SequenceFileAsBinaryOutputFormat.class\n    && FileOutputFormat.getCompressOutput(job)\n    && SequenceFileOutputFormat.getOutputCompressionType(job) == CompressionType.RECORD) {\n  throw new IllegalArgumentException(\n      \"SequenceFileAsBinaryOutputFormat supports only BLOCK or NONE compression\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  job.submit();\n} catch (InvalidJobConfException e) {\n  // checkOutputSpecs failed at submit time; fix compression type to BLOCK and resubmit\n}","preventionTips":["Default new binary-output jobs to CompressionType.BLOCK.","Do not share one compression template across output formats.","Unit-test job assembly by calling outputFormat.checkOutputSpecs() before submission in CI."],"tags":["hadoop","mapreduce","output-format","compression","sequencefile","job-submission"],"backgroundTag":"unsupported-compression-type","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}