{"record":{"id":"013941af85e0146c","repo":"apache/hadoop","slug":"compression-option-provided-does-not-match-the-fil","errorCode":null,"errorMessage":"Compression option provided does not match the file","messagePattern":"Compression option provided does not match the file","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java","lineNumber":1162,"sourceCode":"                  reader.getVersion());\n            }\n\n            if (metadataOption != null) {\n              LOG.info(\"MetaData Option is ignored during append\");\n            }\n            metadataOption = (MetadataOption) SequenceFile.Writer\n                .metadata(reader.getMetadata());\n\n            CompressionOption readerCompressionOption = new CompressionOption(\n                reader.getCompressionType(), reader.getCompressionCodec());\n\n            // Codec comparison will be ignored if the compression is NONE\n            if (readerCompressionOption.value != compressionTypeOption.value\n                || (readerCompressionOption.value != CompressionType.NONE\n                    && readerCompressionOption.codec\n                        .getClass() != compressionTypeOption.codec\n                            .getClass())) {\n              throw new IllegalArgumentException(\n                  \"Compression option provided does not match the file\");\n            }\n\n            sync = reader.getSync();\n\n          } finally {\n            reader.close();\n          }\n\n          out = fs.append(p, bufferSize, progress);\n          this.appendMode = true;\n        } else {\n          out = fs\n              .create(p, true, bufferSize, replication, blockSize, progress);\n        }\n      } else {\n        out = streamOption.getValue();\n      }","sourceCodeStart":1144,"sourceCodeEnd":1180,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java#L1144-L1180","documentation":"In appendIfExists mode the compression stored in the existing file's header must match the CompressionOption passed to createWriter: the CompressionType values must be equal, and unless the type is NONE the codec classes must be identical. Any difference throws this IllegalArgumentException, keeping the appended records decodable with the header's declared codec.","triggerScenarios":"Appending with Writer.compression(CompressionType.BLOCK, new GzipCodec()) when the header says RECORD or NONE; appending SnappyCodec data to a DefaultCodec file; appending with no compression option at all when the file header declares compression (type mismatch NONE vs recorded type).","commonSituations":"Cluster compression defaults changed between runs (mapreduce.map.output.compress.codec) so a rerun appends with a different codec; tools that hardcode compression onto paths written by another tool; codec class relocation after an upgrade changing class identity.","solutions":["Read the file's compression via SequenceFile.Reader.getCompressionType()/getCompressionCodec() and pass exactly that combination to createWriter","Do not guess: omit the compression option only if you have verified the header is uncompressed","If new compression is actually required, write to a new path rather than appending"],"exampleFix":"// before\nWriter w = SequenceFile.createWriter(conf, Writer.file(p), Writer.appendIfExists(true),\n    Writer.compression(CompressionType.BLOCK), ...); // file header says RECORD/DefaultCodec\n\n// after: mirror the header\ntry (SequenceFile.Reader probe = new SequenceFile.Reader(conf,\n    Reader.file(p), new Reader.OnlyHeaderOption())) {\n  CompressionType t = probe.getCompressionType();\n  CompressionCodec c = t == CompressionType.NONE ? null : probe.getCompressionCodec();\n  Writer w = SequenceFile.createWriter(conf, Writer.file(p), Writer.appendIfExists(true),\n      Writer.compression(t, c), ...);\n}","handlingStrategy":"validation","validationCode":"try (SequenceFile.Reader probe = new SequenceFile.Reader(conf,\n    SequenceFile.Reader.file(p), new SequenceFile.Reader.OnlyHeaderOption())) {\n  CompressionType t = probe.getCompressionType();\n  CompressionCodec c = (t == CompressionType.NONE) ? null : probe.getCompressionCodec();\n  // pass Writer.compression(t, c) to createWriter so it mirrors the header\n}","typeGuard":null,"tryCatchPattern":"try {\n  w = SequenceFile.createWriter(conf, opts);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"Compression option\")) {\n    // re-read header compression and rebuild options to match\n  } else { throw e; }\n}","preventionTips":["Derive compression settings from the target file's header when appending, never from job defaults","Pin codec versions across upgrades; class identity is part of the check"],"tags":["hadoop","sequence-file","append","compression","codec-mismatch"],"backgroundTag":"compression-config-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}