apache/hadoop · critical · IOException

Invalid operation read {}

Error message

Invalid operation read {}

What it means

applyEditLogOp hit an opcode with no case in its replay switch and throws IOException('Invalid operation read <opCode>'). Either the decoded opcode is garbage from a corrupt stream, or the log was written by a newer Hadoop whose opcodes this NameNode does not understand: the classic downgrade failure.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java:1056

    case OP_DISABLE_ERASURE_CODING_POLICY:
      DisableErasureCodingPolicyOp disableOp =
          (DisableErasureCodingPolicyOp) op;
      fsNamesys.getErasureCodingPolicyManager().disablePolicy(
          disableOp.getEcPolicy());
      if (toAddRetryCache) {
        fsNamesys.addCacheEntry(op.rpcClientId, op.rpcCallId);
      }
      break;
    case OP_REMOVE_ERASURE_CODING_POLICY:
      RemoveErasureCodingPolicyOp removeOp = (RemoveErasureCodingPolicyOp) op;
      fsNamesys.getErasureCodingPolicyManager().removePolicy(
          removeOp.getEcPolicy());
      if (toAddRetryCache) {
        fsNamesys.addCacheEntry(op.rpcClientId, op.rpcCallId);
      }
      break;
    default:
      throw new IOException("Invalid operation read " + op.opCode);
    }
    return inodeId;
  }
  
  private static String formatEditLogReplayError(EditLogInputStream in,
      long recentOpcodeOffsets[], long txid) {
    StringBuilder sb = new StringBuilder();
    sb.append("Error replaying edit log at offset " + in.getPosition())
        .append(".  Expected transaction ID was ").append(txid);
    if (recentOpcodeOffsets[0] != -1) {
      Arrays.sort(recentOpcodeOffsets);
      sb.append("\nRecent opcode offsets:");
      for (long offset : recentOpcodeOffsets) {
        if (offset != -1) {
          sb.append(' ').append(offset);
        }
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Run the NameNode binary that matches or is newer than the version that wrote the logs; never downgrade across layout-version changes
2. If a downgrade is required, follow the documented rollback procedure and restore the pre-upgrade name directory snapshot instead of replaying new edits with old code
3. Inspect the segment with 'hdfs oev' to identify the offending opcode and txid
4. Restore a consistent fsimage plus edits from backup if the segment is corrupt

Example fix

# before: old binaries replaying newer logs
export HADOOP_PREFIX=/opt/hadoop-2.7 && hdfs --daemon start namenode   # fails

# after: replay with binaries at or above the writer's version, or roll back properly
export HADOOP_PREFIX=/opt/hadoop-3.1 && hdfs --daemon start namenode
# or: restore pre-upgrade name dirs from the rollback snapshot

// refuse to replay logs newer than this binary understands
// (layout versions grow more negative over time)
int logLayout = /* read from the segment header via hdfs oev or EditLogFileInputStream */;
if (logLayout < HdfsServerConstants.NAMENODE_LAYOUT_VERSION) {
  throw new IllegalStateException("Edit log layout " + logLayout
      + " is newer than this NameNode understands; replay it with newer binaries");
}

try {
  loader.loadFSEdits(storage, 0);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Invalid operation read")) {
    // unknown opcode: this binary is older than the log writer.
    // stop the downgrade; restore the pre-upgrade name dirs instead
  }
  throw e;
}

When it happens

Trigger: Starting an older NameNode binary against edit logs containing opcodes introduced later (downgrade after an upgrade); corrupted opcode bytes; replaying segments copied from a cluster on a newer release.

Common situations: Downgrade done by swapping JARs without the documented rollback procedure; edit segments copied between clusters running different versions; media corruption.

Related errors

• Read invalid opcode {opCode}
• Error replaying edit log at offset {}. Expected transaction
• The layout version {} supports inodeId but gave bogus inodeI
• Mismatched block IDs or generation stamps for the old last b
• Mismatched block IDs or generation stamps, attempting to rep

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a5b6ea5fd4bbbdc9. Report an issue: GitHub.