apache/hadoop · error · IllegalArgumentException

Key/value class provided does not match the file

Error message

Key/value class provided does not match the file

What it means

With Writer.appendIfExists(true), createWriter opens the existing sequence file's header and compares its stored key/value classes (by class identity) against Writer.keyClass(...)/Writer.valueClass(...). Any difference throws this IllegalArgumentException instead of appending records of a foreign type and producing an unreadable file.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:1138

        short replication = replicationOption == null ? 
          fs.getDefaultReplication(p) :
          (short) replicationOption.getValue();
        long blockSize = blockSizeOption == null ? fs.getDefaultBlockSize(p) :
          blockSizeOption.getValue();
        Progressable progress = progressOption == null ? null :
          progressOption.getValue();

        if (appendIfExistsOption != null && appendIfExistsOption.getValue()
            && fs.exists(p)) {

          // Read the file and verify header details
          SequenceFile.Reader reader = new SequenceFile.Reader(conf,
              SequenceFile.Reader.file(p), new Reader.OnlyHeaderOption());
          try {

            if (keyClassOption.getValue() != reader.getKeyClass()
                || valueClassOption.getValue() != reader.getValueClass()) {
              throw new IllegalArgumentException(
                  "Key/value class provided does not match the file");
            }

            if (reader.getVersion() != VERSION[3]) {
              throw new VersionMismatchException(VERSION[3],
                  reader.getVersion());
            }

            if (metadataOption != null) {
              LOG.info("MetaData Option is ignored during append");
            }
            metadataOption = (MetadataOption) SequenceFile.Writer
                .metadata(reader.getMetadata());

            CompressionOption readerCompressionOption = new CompressionOption(
                reader.getCompressionType(), reader.getCompressionCodec());

            // Codec comparison will be ignored if the compression is NONE

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the file's real classes with SequenceFile.Reader.getKeyClass()/getValueClass() and pass exactly those to the writer
2. If the schema intentionally changed, write to a new path instead of appending
3. Rename or delete the old file if it is stale and start a fresh one

Example fix

// before
Writer w = SequenceFile.createWriter(conf, Writer.file(p), Writer.appendIfExists(true),
    Writer.keyClass(LongWritable.class), Writer.valueClass(Text.class)); // file holds Text/IntWritable

// after
try (SequenceFile.Reader probe = new SequenceFile.Reader(conf,
    Reader.file(p), new Reader.OnlyHeaderOption())) {
  Writer w = SequenceFile.createWriter(conf, Writer.file(p), Writer.appendIfExists(true),
      Writer.keyClass(probe.getKeyClass()), Writer.valueClass(probe.getValueClass()));
}

try (SequenceFile.Reader probe = new SequenceFile.Reader(conf,
    SequenceFile.Reader.file(p), new SequenceFile.Reader.OnlyHeaderOption())) {
  if (kc != probe.getKeyClass() || vc != probe.getValueClass()) {
    throw new IllegalArgumentException("file holds " + probe.getKeyClass() + "/" + probe.getValueClass()
        + " but caller passed " + kc + "/" + vc);
  }
}

try {
  w = SequenceFile.createWriter(conf, opts);
} catch (IllegalArgumentException e) {
  if ("Key/value class provided does not match the file".equals(e.getMessage())) {
    // schema drift: write to a new path instead of appending
  } else { throw e; }
}

When it happens

Trigger: SequenceFile.createWriter(conf, Writer.file(existingSeqFile), Writer.appendIfExists(true), Writer.keyClass(A.class), Writer.valueClass(B.class)) where the header holds other classes; also triggered by passing a subclass where the header stores the superclass (comparison is !=, not isAssignableFrom).

Common situations: Types evolved between runs (Text keys changed to LongWritable) while output paths stayed fixed; multiple jobs with different schemas writing the same path; refactoring that replaced a Writable with its subclass.

Related errors

• A record version mismatch occurred. Expecting v{}, found v{}
• Compression option provided does not match the file
• wrong key class: {} is not {}
• wrong value class: {} is not {}
• negative length keys not allowed: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6768de30f0c60db6. Report an issue: GitHub.