apache/hadoop · error · VersionMismatchException

A record version mismatch occurred. Expecting v{}, found v{}

Error message

A record version mismatch occurred. Expecting v{}, found v{}

What it means

In appendIfExists mode the existing file must have been written with sequence-file version VERSION[3] (the append-capable header revision); the writer opens a Reader, compares reader.getVersion() against it and throws VersionMismatchException (expected vs found version bytes) for anything older. Old headers lack the layout guarantees appending relies on, so the writer refuses rather than corrupt them.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:1143

        Progressable progress = progressOption == null ? null :
          progressOption.getValue();

        if (appendIfExistsOption != null && appendIfExistsOption.getValue()
            && fs.exists(p)) {

          // Read the file and verify header details
          SequenceFile.Reader reader = new SequenceFile.Reader(conf,
              SequenceFile.Reader.file(p), new Reader.OnlyHeaderOption());
          try {

            if (keyClassOption.getValue() != reader.getKeyClass()
                || valueClassOption.getValue() != reader.getValueClass()) {
              throw new IllegalArgumentException(
                  "Key/value class provided does not match the file");
            }

            if (reader.getVersion() != VERSION[3]) {
              throw new VersionMismatchException(VERSION[3],
                  reader.getVersion());
            }

            if (metadataOption != null) {
              LOG.info("MetaData Option is ignored during append");
            }
            metadataOption = (MetadataOption) SequenceFile.Writer
                .metadata(reader.getMetadata());

            CompressionOption readerCompressionOption = new CompressionOption(
                reader.getCompressionType(), reader.getCompressionCodec());

            // Codec comparison will be ignored if the compression is NONE
            if (readerCompressionOption.value != compressionTypeOption.value
                || (readerCompressionOption.value != CompressionType.NONE
                    && readerCompressionOption.codec
                        .getClass() != compressionTypeOption.codec
                            .getClass())) {

View on GitHub (pinned to 2add963021)

Solutions

1. Read the old file with SequenceFile.Reader and copy its records into a new file created by the current writer, then append to that file
2. Or simply write a new output part-file instead of appending to the legacy one
3. Standardize writer and reader on the same Hadoop version so headers are uniform

Example fix

// before
Writer w = SequenceFile.createWriter(conf, Writer.file(oldFile), Writer.appendIfExists(true), ...); // VersionMismatchException

// after: rewrite to a fresh file with the current writer, then append
try (SequenceFile.Reader in = new SequenceFile.Reader(conf, Reader.file(oldFile));
     SequenceFile.Writer out = SequenceFile.createWriter(conf, Writer.file(newFile),
         Writer.keyClass(in.getKeyClass()), Writer.valueClass(in.getValueClass()))) {
  Object k = ReflectionUtils.newInstance(in.getKeyClass(), conf);
  Object v = ReflectionUtils.newInstance(in.getValueClass(), conf);
  while (in.next(k, v)) { out.append(k, v); }
}

// sequence file header: 3 magic bytes + 1 version byte; append needs version 6
try (FSDataInputStream in = fs.open(p)) {
  byte[] hdr = new byte[4];
  in.readFully(hdr);
  if (hdr[3] != 6) {
    throw new IOException("file version " + hdr[3] + " does not support append; rewrite it first");
  }
}

try {
  w = SequenceFile.createWriter(conf, opts); // appendIfExists(true)
} catch (VersionMismatchException e) {
  // expected vs found version in the message; rewrite the file with the current writer
}

When it happens

Trigger: Appending (Writer.appendIfExists(true)) to a sequence file produced by an older Hadoop writer with a pre-append version byte, or to a file whose header was hand-edited/damaged so the version byte reads as something unexpected.

Common situations: Appending to data files archived from an old cluster; mixed-version pipelines where an old writer produced the file; tutorials/data generated years earlier being reused as append targets.

Related errors

• Key/value class provided does not match the file
• Compression option provided does not match the file
• wrong key class: {} is not {}
• wrong value class: {} is not {}
• negative length keys not allowed: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ecaf32dfb3809878. Report an issue: GitHub.