What causes "Not replicated yet: " in apache/hadoop?

Let the standard DFSOutputStream handle it: it retries NotReplicatedYetException with backoff and usually self-heals once datanodes report the block

apache/hadoop · error · NotReplicatedYetException

Not replicated yet: <src>

Error message

Not replicated yet: <src>

What it means

Before allocating a new block for a file under construction, the NameNode requires every block except the last to be at least minimally replicated (dfs.namenode.replication.min, default 1). analyzeFileState calls checkFileProgress and throws NotReplicatedYetException when the penultimate block has not reached that level: the client is asking for the next block faster than datanodes have reported the previous one.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java:183

      ExtendedBlock previous, LocatedBlock[] onRetryBlock) throws IOException {
    final long blockSize;
    final short numTargets;
    final byte storagePolicyID;
    String clientMachine;
    final BlockType blockType;

    INodesInPath iip = fsn.dir.resolvePath(pc, src, fileId);
    FileState fileState = analyzeFileState(fsn, iip, fileId, clientName,
                                           previous, onRetryBlock);
    if (onRetryBlock[0] != null && onRetryBlock[0].getLocations().length > 0) {
      // This is a retry. No need to generate new locations.
      // Use the last block if it has locations.
      return null;
    }

    final INodeFile pendingFile = fileState.inode;
    if (!fsn.checkFileProgress(src, pendingFile, false)) {
      throw new NotReplicatedYetException("Not replicated yet: " + src);
    }
    if (pendingFile.getBlocks().length >= fsn.maxBlocksPerFile) {
      throw new IOException("File has reached the limit on maximum number of"
          + " blocks (" + DFSConfigKeys.DFS_NAMENODE_MAX_BLOCKS_PER_FILE_KEY
          + "): " + pendingFile.getBlocks().length + " >= "
          + fsn.maxBlocksPerFile);
    }
    blockSize = pendingFile.getPreferredBlockSize();
    clientMachine = pendingFile.getFileUnderConstructionFeature()
        .getClientMachine();
    blockType = pendingFile.getBlockType();
    ErasureCodingPolicy ecPolicy = null;
    if (blockType == BlockType.STRIPED) {
      ecPolicy =
          FSDirErasureCodingOp.unprotectedGetErasureCodingPolicy(fsn, iip);
      numTargets = (short) (ecPolicy.getSchema().getNumDataUnits()
          + ecPolicy.getSchema().getNumParityUnits());
    } else {

View on GitHub (pinned to 2add963021)

Solutions

Let the standard DFSOutputStream handle it: it retries NotReplicatedYetException with backoff and usually self-heals once datanodes report the block
Check cluster health: 'hdfs dfsadmin -report' for live nodes and 'hdfs fsck <path>' for under-replicated blocks; fix dead disks or full datanodes
Ensure enough healthy datanodes exist for the file's replication factor before large writes
Only as a last resort, lower dfs.namenode.replication.min, accepting weaker durability

Example fix

// before: raw addBlock call fails fast
LocatedBlock lb = namenode.addBlock(src, clientName, previous, fileId, null, 0);

// after: honor the transient backoff signal
long delay = 500L;
for (int i = 0; i < 30; i++) {
  try {
    lb = namenode.addBlock(src, clientName, previous, fileId, null, 0);
    break;
  } catch (NotReplicatedYetException e) {
    Thread.sleep(delay);
    delay = Math.min(delay * 2, 10_000L);
  }
}

Defensive patterns

Strategy: retry

Validate before calling

DistributedFileSystem dfs = (DistributedFileSystem) fs;
int live = dfs.getDataNodeStats(DatanodeReportType.LIVE).length;
if (live < fileReplication) {
  throw new IOException("Only " + live + " live datanodes for replication " + fileReplication);
}

Try / catch

long delay = 500L;
while (true) {
  try {
    return namenode.addBlock(src, clientName, previous, fileId, null, 0);
  } catch (NotReplicatedYetException e) {
    Thread.sleep(delay);
    delay = Math.min(delay * 2, 10_000L);
  }
}

Prevention

Use DFSOutputStream instead of raw ClientProtocol; it already retries this exception with backoff
Keep enough healthy datanodes for the replication factor during bulk writes
Avoid rolling datanode restarts while large writes are in flight

When it happens

Trigger: ClientProtocol.addBlock() for the next block while the previous block still has fewer than min-replication live replicas: right after a pipeline error, when datanodes are dead, full, or restarting, or when block reports have not arrived yet.

Common situations: Dead or overloaded datanodes; cluster too small for the file's replication factor; custom clients calling addBlock in a tight retry loop; rolling datanode restarts during writes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c21105e80fcd92bd. Report an issue: GitHub.