apache/hadoop · error · IllegalArgumentException

Compression option provided does not match the file

Error message

Compression option provided does not match the file

What it means

In appendIfExists mode the compression stored in the existing file's header must match the CompressionOption passed to createWriter: the CompressionType values must be equal, and unless the type is NONE the codec classes must be identical. Any difference throws this IllegalArgumentException, keeping the appended records decodable with the header's declared codec.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:1162

                  reader.getVersion());
            }

            if (metadataOption != null) {
              LOG.info("MetaData Option is ignored during append");
            }
            metadataOption = (MetadataOption) SequenceFile.Writer
                .metadata(reader.getMetadata());

            CompressionOption readerCompressionOption = new CompressionOption(
                reader.getCompressionType(), reader.getCompressionCodec());

            // Codec comparison will be ignored if the compression is NONE
            if (readerCompressionOption.value != compressionTypeOption.value
                || (readerCompressionOption.value != CompressionType.NONE
                    && readerCompressionOption.codec
                        .getClass() != compressionTypeOption.codec
                            .getClass())) {
              throw new IllegalArgumentException(
                  "Compression option provided does not match the file");
            }

            sync = reader.getSync();

          } finally {
            reader.close();
          }

          out = fs.append(p, bufferSize, progress);
          this.appendMode = true;
        } else {
          out = fs
              .create(p, true, bufferSize, replication, blockSize, progress);
        }
      } else {
        out = streamOption.getValue();
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Read the file's compression via SequenceFile.Reader.getCompressionType()/getCompressionCodec() and pass exactly that combination to createWriter
2. Do not guess: omit the compression option only if you have verified the header is uncompressed
3. If new compression is actually required, write to a new path rather than appending

Example fix

// before
Writer w = SequenceFile.createWriter(conf, Writer.file(p), Writer.appendIfExists(true),
    Writer.compression(CompressionType.BLOCK), ...); // file header says RECORD/DefaultCodec

// after: mirror the header
try (SequenceFile.Reader probe = new SequenceFile.Reader(conf,
    Reader.file(p), new Reader.OnlyHeaderOption())) {
  CompressionType t = probe.getCompressionType();
  CompressionCodec c = t == CompressionType.NONE ? null : probe.getCompressionCodec();
  Writer w = SequenceFile.createWriter(conf, Writer.file(p), Writer.appendIfExists(true),
      Writer.compression(t, c), ...);
}

try (SequenceFile.Reader probe = new SequenceFile.Reader(conf,
    SequenceFile.Reader.file(p), new SequenceFile.Reader.OnlyHeaderOption())) {
  CompressionType t = probe.getCompressionType();
  CompressionCodec c = (t == CompressionType.NONE) ? null : probe.getCompressionCodec();
  // pass Writer.compression(t, c) to createWriter so it mirrors the header
}

try {
  w = SequenceFile.createWriter(conf, opts);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("Compression option")) {
    // re-read header compression and rebuild options to match
  } else { throw e; }
}

When it happens

Trigger: Appending with Writer.compression(CompressionType.BLOCK, new GzipCodec()) when the header says RECORD or NONE; appending SnappyCodec data to a DefaultCodec file; appending with no compression option at all when the file header declares compression (type mismatch NONE vs recorded type).

Common situations: Cluster compression defaults changed between runs (mapreduce.map.output.compress.codec) so a rerun appends with a different codec; tools that hardcode compression onto paths written by another tool; codec class relocation after an upgrade changing class identity.

Related errors

• Key/value class provided does not match the file
• A record version mismatch occurred. Expecting v{}, found v{}
• wrong key class: {} is not {}
• wrong value class: {} is not {}
• negative length keys not allowed: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/013941af85e0146c. Report an issue: GitHub.