apache/hadoop · error · IOException

"append: lastBlock=" + lastBlock + " of src=" + path + " is…

Error message

"append: lastBlock=" + lastBlock + " of src=" + path + " is not sufficiently replicated yet."

What it means

append requires the last complete block to be at least minimally replicated before the file may be extended; blockManager.isSufficientlyReplicated(lastBlock) returned false, so the NameNode throws IOException 'not sufficiently replicated yet'. Unlike the COMMITTED case this is about live replica count, not block finalization state.

Solutions

1. Wait and retry the append - re-replication usually restores sufficient replication on its own.
2. Run 'hdfs fsck <path> -files -blocks -locations' to see the last block's replica locations and whether replicas are missing or merely pending.
3. Restore dead DataNodes or let under-replication processing fix the block; if replicas are permanently lost, the file needs manual recovery (fsck -repair or re-copy from source).

for (int attempt = 0; attempt < 10; attempt++) {
  try {
    return dfs.append(path, bufferSize);
  } catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("not sufficiently replicated")) {
      Thread.sleep(2000L); // wait for re-replication
      continue;
    }
    throw e;
  }
}

When it happens

Trigger: append() when the file's last block has fewer live replicas than the minimum (dfs.namenode.replication.min): DataNodes holding replicas died or were decommissioned, or replication has not caught up after a node failure.

Common situations: Clusters with DataNodes down or in maintenance; files whose only replicas of recent blocks were on one lost node; small clusters where a single dead DataNode drops last blocks below minimum replication.

Related errors

• "append: lastBlock=" + lastBlock + " of src=" + path + " is…
• Append on block returned a replica of state ; expected RBW
• Cannot append to a replica with unexpected generation stamp
• Cannot append to an unfinalized replica
• Corrupted replica with a length of expected length is

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d17148f9604a4bda. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java:139

      if (lpPolicy != null && lpPolicy.getId() == file.getStoragePolicyID()) {
        throw new UnsupportedOperationException(
            "Cannot append to lazy persist file " + path);
      }
      // Opening an existing file for append - may need to recover lease.
      fsn.recoverLeaseInternal(RecoverLeaseOp.APPEND_FILE, iip, path, holder,
          clientMachine, false);

      final BlockInfo lastBlock = file.getLastBlock();
      // Check that the block has at least minimum replication.
      if (lastBlock != null) {
        if (lastBlock.getBlockUCState() == BlockUCState.COMMITTED) {
          throw new RetriableException(
              new NotReplicatedYetException("append: lastBlock="
                  + lastBlock + " of src=" + path
                  + " is COMMITTED but not yet COMPLETE."));
        } else if (lastBlock.isComplete()
          && !blockManager.isSufficientlyReplicated(lastBlock)) {
          throw new IOException("append: lastBlock=" + lastBlock + " of src="
              + path + " is not sufficiently replicated yet.");
        }
      }
      lb = prepareFileForAppend(fsn, iip, holder, clientMachine, newBlock,
          true, logRetryCache);
    } catch (IOException ie) {
      NameNode.stateChangeLog
          .warn("DIR* NameSystem.append: " + ie.getMessage());
      throw ie;
    } finally {
      fsd.writeUnlock();
    }

    HdfsFileStatus stat =
        FSDirStatAndListingOp.getFileInfo(fsd, iip, false, false);
    if (lb != null) {
      NameNode.stateChangeLog.debug(
          "DIR* NameSystem.appendFile: file {} for {} at {} block {} block"

View on GitHub (pinned to 2add963021)