apache/hadoop · warning · RetriableException

"append: lastBlock=" + lastBlock + " of src=" + path + " is

Error message

"append: lastBlock=" + lastBlock + " of src=" + path + " is COMMITTED but not yet COMPLETE."

What it means

The file's last block is in COMMITTED state: the writer requested finalization but the NameNode has not yet seen COMPLETE reports from enough DataNodes, so append cannot safely extend or add a block after it. The NameNode wraps NotReplicatedYetException in RetriableException, explicitly telling the client to retry the same append later instead of failing.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java:133

                + CreateFlag.NEW_BLOCK + " create flag while appending file.");
      }

      BlockManager blockManager = fsd.getBlockManager();
      final BlockStoragePolicy lpPolicy = blockManager
          .getStoragePolicy("LAZY_PERSIST");
      if (lpPolicy != null && lpPolicy.getId() == file.getStoragePolicyID()) {
        throw new UnsupportedOperationException(
            "Cannot append to lazy persist file " + path);
      }
      // Opening an existing file for append - may need to recover lease.
      fsn.recoverLeaseInternal(RecoverLeaseOp.APPEND_FILE, iip, path, holder,
          clientMachine, false);

      final BlockInfo lastBlock = file.getLastBlock();
      // Check that the block has at least minimum replication.
      if (lastBlock != null) {
        if (lastBlock.getBlockUCState() == BlockUCState.COMMITTED) {
          throw new RetriableException(
              new NotReplicatedYetException("append: lastBlock="
                  + lastBlock + " of src=" + path
                  + " is COMMITTED but not yet COMPLETE."));
        } else if (lastBlock.isComplete()
          && !blockManager.isSufficientlyReplicated(lastBlock)) {
          throw new IOException("append: lastBlock=" + lastBlock + " of src="
              + path + " is not sufficiently replicated yet.");
        }
      }
      lb = prepareFileForAppend(fsn, iip, holder, clientMachine, newBlock,
          true, logRetryCache);
    } catch (IOException ie) {
      NameNode.stateChangeLog
          .warn("DIR* NameSystem.append: " + ie.getMessage());
      throw ie;
    } finally {
      fsd.writeUnlock();
    }

View on GitHub (pinned to 2add963021)

Solutions

  1. Retry the append with short backoff - RetriableException is designed for retry and typically clears within seconds once the block reaches COMPLETE.
  2. Ensure the previous writer fully closes its output stream before the next stage starts appending.
  3. If it persists, check DataNode liveness ('hdfs dfsadmin -report') and NameNode metrics for pending block-completion backlogs.
Defensive patterns

Strategy: retry

Try / catch

for (int attempt = 0; attempt < 5; attempt++) {
  try {
    return dfs.append(path, bufferSize);
  } catch (RemoteException re) {
    if ("org.apache.hadoop.ipc.RetriableException".equals(re.getClassName())) {
      Thread.sleep(500L << attempt); // exponential backoff, block is finalizing
      continue;
    }
    throw re;
  }
}
throw new IOException("append still retriable after retries: " + path);

Prevention

When it happens

Trigger: Calling append immediately after another writer closes or hflushes the same file while block finalization is still in flight; appending right after lease recovery; DataNodes slow to process blockReceivedAndDeleted reports.

Common situations: Two pipeline stages appending to one file back-to-back; append after recovering the lease of a dead client; overloaded or GC-pausing DataNodes delaying incremental block reports.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f3a99fad4c96b7ec. Report an issue: GitHub.