What causes "Failed to read fields from SetAclOp" in apache/hadoop?

Run 'hdfs namenode -recover' to skip the truncated tail record, then immediately checkpoint

apache/hadoop · error · java.io.IOException

Failed to read fields from SetAclOp

Error message

Failed to read fields from SetAclOp

What it means

SetAclOp.readFields() parses its record body as a length-delimited protobuf (AclEditLogProto.parseDelimitedFrom). parseDelimitedFrom returns null exactly when EOF is reached before the message's length prefix, which this code converts into an IOException: the segment ends in the middle of an OP_SET_ACL record.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:4422

    SetAclOp() {
      super(OP_SET_ACL);
    }

    static SetAclOp getInstance(OpInstanceCache cache) {
      return cache.get(OP_SET_ACL);
    }

    @Override
    void resetSubFields() {
      aclEntries = null;
      src = null;
    }

    @Override
    void readFields(DataInputStream in, int logVersion) throws IOException {
      AclEditLogProto p = AclEditLogProto.parseDelimitedFrom(in);
      if (p == null) {
        throw new IOException("Failed to read fields from SetAclOp");
      }
      src = p.getSrc();
      aclEntries = PBHelperClient.convertAclEntry(p.getEntriesList());
    }

    @Override
    public void writeFields(DataOutputStream out) throws IOException {
      AclEditLogProto.Builder b = AclEditLogProto.newBuilder();
      if (src != null)
        b.setSrc(src);
      b.addAllEntries(PBHelperClient.convertAclEntryProto(aclEntries));
      b.build().writeDelimitedTo(out);
    }

    @Override
    protected void toXml(ContentHandler contentHandler) throws SAXException {
      XMLUtils.addSaxString(contentHandler, "SRC", src);
      appendAclEntriesToXml(contentHandler, aclEntries);

View on GitHub (pinned to 2add963021)

Solutions

Run 'hdfs namenode -recover' to skip the truncated tail record, then immediately checkpoint
Inspect the segment with 'hdfs offlineEditsViewer -i <edits> -o out.xml' to confirm where it truncates
Fetch a complete copy of the same segment from a healthy JournalNode (QJM majority) and replace the short file
If only the local copy exists and is truncated, discard it and recover from the last fsimage plus valid journals; also fix the root cause (disk space, network) before restarting

Example fix

# before
hdfs namenode   # IOException: Failed to read fields from SetAclOp
# after
hdfs namenode -recover   # skip the truncated tail, then: hdfs dfsadmin -saveNamespace

Defensive patterns

Strategy: try-catch

Validate before calling

# verify every segment is complete before replay/bootstrap
hdfs offlineEditsViewer -i edits_inprogress_0000000000000001234 -o /dev/null
# a truncated protobuf tail fails here instead of during NameNode start

Try / catch

try {
  FSEditLogOp op = reader.readOp();
} catch (IOException e) {
  if (e.getMessage().contains("Failed to read fields from SetAclOp")) {
    // segment truncated mid-OP_SET_ACL: recover with 'hdfs namenode -recover'
    // or refetch the complete segment from a healthy JournalNode
  } else { throw e; }
}

Prevention

Keep journal disks below capacity alerts; disk-full truncates in-flight records
After a crash, validate edits_inprogress with offlineEditsViewer before restart
Run QJM so a complete copy of every segment always exists elsewhere
Verify network health between NN/JN/SNN — truncated transfers produce short segments

When it happens

Trigger: Replaying an edits_inprogress or finalized segment truncated mid-OP_SET_ACL: NameNode killed or the journal disk filled during the write, a short HTTP/JournalNode transfer delivered an incomplete file, or bit rot consumed the tail.

Common situations: Unclean shutdown followed by restart; full journal disks; standby bootstrap or offlineEditsViewer pulling a truncated segment over a flaky network; corrupted transfers between JNs.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/644291eafe33a02e. Report an issue: GitHub.