apache/hadoop · error · IOException

Counters version mismatch, expected ${groupFactory.version()

Error message

Counters version mismatch, expected ${groupFactory.version()} got ${version}

What it means

org.apache.hadoop.mapreduce.Counters (via AbstractCounters) is Hadoop Writable-serializable and writes a version number (from the GroupFactory) at the head of the byte stream. readFields throws IOException when the wire version does not equal the local groupFactory.version(), meaning the stream was written by a different counters implementation. This is a hard compatibility barrier: Writable Counters are not a stable cross-version exchange format.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java:288

        group.write(out);
      }
    }
    if (writeAllCounters) {
      WritableUtils.writeVInt(out, groups.size());
      for (G group : groups.values()) {
        Text.writeString(out, group.getName());
        group.write(out);
      }
    } else {
      WritableUtils.writeVInt(out, 0);
    }
  }

  @Override
  public synchronized void readFields(DataInput in) throws IOException {
    int version = WritableUtils.readVInt(in);
    if (version != groupFactory.version()) {
      throw new IOException("Counters version mismatch, expected "+
          groupFactory.version() +" got "+ version);
    }
    int numFGroups = WritableUtils.readVInt(in);
    fgroups.clear();
    GroupType[] groupTypes = GroupType.values();
    while (numFGroups-- > 0) {
      GroupType groupType = groupTypes[WritableUtils.readVInt(in)];
      G group;
      switch (groupType) {
        case FILESYSTEM: // with nothing
          group = groupFactory.newFileSystemGroup();
          break;
        case FRAMEWORK:  // with group id
          group = groupFactory.newFrameworkGroup(WritableUtils.readVInt(in));
          break;
        default: // Silence dumb compiler, as it would've thrown earlier
          throw new IOException("Unexpected counter group type: "+ groupType);
      }

View on GitHub (pinned to 2add963021)

Solutions

Deserialize the counters with the exact same Hadoop build (and same GroupFactory) that serialized them.
For cross-version exchange, use stable representations instead: the JobHistoryServer REST API, 'mapreduce job -status' output, or JSON from the history viewer.
If you subclass AbstractCounters, keep groupFactory.version() aligned on both ends and change it only with a matching read path.
Re-run/recompute the counters rather than replaying old blobs after an upgrade.

Defensive patterns

Strategy: try-catch

Try / catch

catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Counters version mismatch")) {
    // stream written by a different counters build: discard, start from new Counters()
    counters = job.newCounters();
  } else { throw e; }
}

Prevention

Pin one Hadoop version on both ends of any channel that carries Writable Counters.
Never persist Writable Counters as a long-term exchange format; use JSON/REST outputs instead.
If subclassing AbstractCounters, keep groupFactory.version() identical across your deployed builds unless you handle migration.

When it happens

Trigger: Counters.readFields() deserializing bytes produced by a different Hadoop major/minor build (version VInt differs), or by a custom AbstractCounters subclass whose GroupFactory reports a different version. Typical paths: RPC between nodes with mismatched Hadoop versions, replaying serialized counters from persisted files or captured events.

Common situations: Rolling upgrades where an old client/task reads counters from a newer cluster; tools that persist Writable Counters blobs and read them back after a Hadoop upgrade; custom counters frameworks overriding the GroupFactory without keeping version() consistent.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f0ef61c17cf599bb. Report an issue: GitHub.