apache/hadoop · error · IOException

Unable to close file because the last block {} does not have

Error message

Unable to close file because the last block {} does not have enough number of replicas.

What it means

completeFile() gives up when the NameNode keeps answering 'not complete' for the final namenode.complete() call because the last block has not reached minimal replication (e.g. the pipeline lost DNs and the block is under-replicated). The client retries with exponential backoff (initial locateFollowingBlock delay, capped) up to dfs.client.block.write.locateFollowingBlock.retries (default 5) and then throws this IOException. The file stays in the NN's filesystem open-for-write under the client's lease until the lease is recovered or the block re-replicates and a later complete succeeds.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:1014

      fileComplete =
          dfsClient.namenode.complete(src, dfsClient.clientName, last, fileId);
      if (!fileComplete) {
        final int hdfsTimeout = conf.getHdfsTimeout();
        if (!dfsClient.clientRunning
            || (hdfsTimeout > 0
                && localstart + hdfsTimeout < Time.monotonicNow())) {
          String msg = "Unable to close file because dfsclient " +
              " was unable to contact the HDFS servers. clientRunning " +
              dfsClient.clientRunning + " hdfsTimeout " + hdfsTimeout;
          DFSClient.LOG.info(msg);
          throw new IOException(msg);
        }
        try (TraceScope scope = dfsClient.getTracer()
            .newScope("DFSOutputStream#completeFile: Retry")) {
          scope.addKVAnnotation("retries left", retries);
          scope.addKVAnnotation("sleeptime (sleeping for)", sleeptime);
          if (retries == 0) {
            throw new IOException("Unable to close file because the last block "
                + last + " does not have enough number of replicas.");
          }
          retries--;
          Thread.sleep(sleeptime);
          sleeptime = calculateDelayForNextRetry(sleeptime, maxSleepTime);
          if (Time.monotonicNow() - localstart > 5000) {
            DFSClient.LOG.info("Could not complete " + src + " retrying...");
          }
        } catch (InterruptedException ie) {
          DFSClient.LOG.warn("Caught exception ", ie);
        }
      }
    }
  }

  @VisibleForTesting
  public void setArtificialSlowdown(long period) {
    getStreamer().setArtificialSlowdown(period);

View on GitHub (pinned to 2add963021)

Solutions

Check and fix DN health (hdfs dfsadmin -report) and wait for the NN to re-replicate the under-replicated block (watch 'hdfs fsck / -list-underreplicated' empty out), then retry close() - or simply let lease recovery finalize the file (soft limit 60s if this client dies, hard limit 1h otherwise).
If you need the file closed now and can afford the current replica count, run 'hdfs debug recoverLease -path <file>' to force finalization, then verify data with 'hdfs fsck <file> -files -blocks'.
Raise dfs.client.block.write.locateFollowingBlock.retries on the client so close() tolerates longer re-replication windows.
For test clusters, keep dfs.replication >= 2 so a single DN restart cannot strand the last block.

Defensive patterns

Strategy: retry

Try / catch

try {
  out.close();
} catch (IOException e) {
  if (!String.valueOf(e.getMessage()).contains("does not have enough number of replicas")) throw e;
  for (int i = 1; i <= 10; i++) {
    Thread.sleep(10_000L);
    if (underReplicatedBlocks(fs, path) == 0) break; // NN finished re-replication
  }
  ((DistributedFileSystem) fs).recoverLease(path); // force completion at current replication
  verifyComplete(path);
}

Prevention

Keep dfs.replication >= 2 so one lost DN cannot strand the last block.
Watch 'hdfs fsck -list-underreplicated' during DN flakiness and let it drain before batch close.
Raise dfs.client.block.write.locateFollowingBlock.retries for close() tolerance.
hflush() before close so data is safe even if completion is deferred to lease recovery.

When it happens

Trigger: close() when one or more DNs in the last block's pipeline died near the end of the write, so the NN cannot count the block as minimally replicated; replication=1 test files whose single DN is slow to block-report or was restarted; appends to files whose last block was already under-replicated before the append.

Common situations: Small/test clusters with dfs.replication=1 and DN restarts; clusters losing DNs (disk/network failures) during large batch writes; long-running writers that finish while the NN still shows stale block reports; the classic job symptom: task fails on close with this message after DN flakiness.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c1261c4e7f61f983. Report an issue: GitHub.