apache/hadoop · error · IOException

invalid metadata header version {version}. Can only handle

Error message

invalid metadata header version {version}.  Can only handle version 1.

What it means

Thrown while constructing a ShortCircuitReplica during an HDFS short-circuit read. With short-circuit reads enabled, the datanode hands the client the replica's block and meta file descriptors, and the client preads the meta file header via BlockMetadataHeader.preadHeader(). This client only understands metadata header version 1 (checksum type plus bytes-per-checksum), so any other version is rejected with an IOException. A non-1 version means the meta file was written in an incompatible format or is corrupt or truncated.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java:131

  /**
   * The monotonic time in nanoseconds at which the replica became evictable, or
   * null if it is not evictable.
   *
   * Protected by the cache lock.
   */
  private Long evictableTimeNs = null;

  public ShortCircuitReplica(ExtendedBlockId key,
      FileInputStream dataStream, FileInputStream metaStream,
      ShortCircuitCache cache, long creationTimeMs, Slot slot) throws IOException {
    this.key = key;
    this.dataStream = dataStream;
    this.metaStream = metaStream;
    this.metaHeader =
          BlockMetadataHeader.preadHeader(metaStream.getChannel());
    if (metaHeader.getVersion() != 1) {
      throw new IOException("invalid metadata header version " +
          metaHeader.getVersion() + ".  Can only handle version 1.");
    }
    this.cache = cache;
    this.creationTimeMs = creationTimeMs;
    this.slot = slot;
  }

  /**
   * Decrement the reference count.
   */
  public void unref() {
    cache.unref(this);
  }

  /**
   * Check if the replica is stale.
   *
   * Must be called with the cache lock held.

View on GitHub (pinned to 2add963021)

Solutions

Run `hdfs fsck <path> -files -blocks -locations` to confirm block/meta health and force re-replication of corrupt replicas
Temporarily set dfs.client.read.shortcircuit=false on the client to fall back to normal datanode-mediated reads and confirm the meta file is at fault
Align client and datanode Hadoop versions; the meta format is version-coupled between them
On the datanode, hexdump the replica's .meta file header (first short must be 1); delete the bad replica so HDFS re-replicates it

Example fix

// before (client config)
<property><name>dfs.client.read.shortcircuit</name><value>true</value></property>

// after - isolate the fault by disabling short-circuit reads
<property><name>dfs.client.read.shortcircuit</name><value>false</value></property>

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate the meta header before accepting a short-circuit replica
try (FileInputStream meta = new FileInputStream(metaFile)) {
  BlockMetadataHeader header = BlockMetadataHeader.preadHeader(meta.getChannel());
  if (header.getVersion() != 1) {
    // refuse short-circuit for this block; use the normal read path
  }
}

Try / catch

try {
  // open/read with dfs.client.read.shortcircuit=true
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("invalid metadata header version")) {
    conf.setBoolean("dfs.client.read.shortcircuit", false); // reopen via normal path
  } else { throw e; }
}

Prevention

Run client and datanode on the same Hadoop release when short-circuit reads are enabled
Schedule periodic hdfs fsck so corrupt replicas are re-replicated before readers hit them
Smoke-test short-circuit reads after every cluster upgrade before rolling the setting out

When it happens

Trigger: dfs.client.read.shortcircuit=true (with a domain socket at dfs.domain.socket.path) and ShortCircuitCache creating a replica whose meta stream preads a header with getVersion() != 1: a truncated meta file, corrupted first bytes on a failing disk, or a meta file produced by a datanode with a different metadata layout.

Common situations: Client and datanode running different Hadoop releases with short-circuit reads enabled; a replica corrupted by disk trouble or a crash; a block/meta pair left inconsistent across an upgrade.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8f763fff88e36785. Report an issue: GitHub.