apache/hadoop · error · IOException

unknown flags set in ModifyCacheDirectiveInfoOp: {}

Error message

unknown flags set in ModifyCacheDirectiveInfoOp: {}

What it means

Cache-directive modify operations are serialized with a 4-bit flag word (0x1 id, 0x2 replication, 0x4 pool, 0x8 expiration). While deserializing a ModifyCacheDirectiveInfoOp, any set bit outside the 0xF mask is rejected: the stream contains fields this build does not understand. The edits were written by a newer release that defined an extra flag, or the data was hand-crafted.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java:565

    CacheDirectiveInfo.Builder builder =
        new CacheDirectiveInfo.Builder();
    builder.setId(readLong(in));
    int flags = in.readInt();
    if ((flags & 0x1) != 0) {
      builder.setPath(new Path(readString(in)));
    }
    if ((flags & 0x2) != 0) {
      builder.setReplication(readShort(in));
    }
    if ((flags & 0x4) != 0) {
      builder.setPool(readString(in));
    }
    if ((flags & 0x8) != 0) {
      builder.setExpiration(
          CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)));
    }
    if ((flags & ~0xF) != 0) {
      throw new IOException("unknown flags set in " +
          "ModifyCacheDirectiveInfoOp: " + flags);
    }
    return builder.build();
  }

  public static CacheDirectiveInfo readCacheDirectiveInfo(Stanza st)
      throws InvalidXmlException {
    CacheDirectiveInfo.Builder builder =
        new CacheDirectiveInfo.Builder();
    builder.setId(Long.parseLong(st.getValue("ID")));
    String path = st.getValueOrNull("PATH");
    if (path != null) {
      builder.setPath(new Path(path));
    }
    String replicationString = st.getValueOrNull("REPLICATION");
    if (replicationString != null) {
      builder.setReplication(Short.parseShort(replicationString));
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Replay the edits with the same or newer Hadoop release that wrote them
2. Do not hand-edit OEV XML; regenerate from the original binary edits
3. If version skew is from a rolling upgrade, finish it so all NNs run the writer's version or later
4. Convert the edits to a loadable form using the newer version's tooling before loading

Example fix

# before: old build rejects newer edits
hdfs oev -i edits_0000000000000001000-0000000000000001500 -p xml -o e.xml
# after: use the writer's release (or newer) to read/convert
/opt/hadoop-3.3.6/bin/hdfs oev -i edits_0000000000000001000-0000000000000001500 -p xml -o e.xml

// reject edits written by a newer layout before replay
NameNodeLayoutVersion_current = HdfsServerConstants.NAMENODE_LAYOUT_VERSION;
if (editsLayoutVersion < HdfsServerConstants.NAMENODE_LAYOUT_VERSION) {
  throw new IOException("edits layout " + editsLayoutVersion
      + " newer than this build supports");
}

catch IOException with 'unknown flags set in ModifyCacheDirectiveInfoOp' during edits replay — abort replay of that file and load it with the writer's Hadoop version instead of skipping records silently.

When it happens

Trigger: Reading edits or OEV XML containing a ModifyCacheDirectiveInfoOp with (flags & ~0xF) != 0 — e.g. replaying an edits file produced by a newer Hadoop release, or loading hand-edited OfflineEditsViewer output.

Common situations: Downgrade while cache-directive edits from a newer release are still in the journal; hand-edited OEV XML round-trips; copying edits across clusters of different versions.

Related errors

• Unknown flag in CachePoolInfo: {}
• Unrecognized section {}
• Edits file is not found in {}
• Invalid block array length: {}
• Unsupported file version {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/13eca403900c449c. Report an issue: GitHub.