apache/hadoop · error · IOException

Unable to close file because dfsclient was unable to contac

Error message

Unable to close file because dfsclient  was unable to contact the HDFS servers. clientRunning {} hdfsTimeout {}

What it means

completeFile() loops calling namenode.complete() until the NN confirms the file is done. Each 'false' answer means the NN could not yet complete the file, and the client bails out with this IOException when either dfsClient.clientRunning is false (the DFSClient is shutting down - e.g. FileSystem.close()/static cache clear, JVM shutdown hook, or an abort) or the client HDFS timeout (hdfsTimeout, 0 = disabled) elapsed since the close started. The message itself is logged at INFO before being thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:1007

    long localstart = Time.monotonicNow();
    final DfsClientConf conf = dfsClient.getConf();
    long sleeptime = conf.getBlockWriteLocateFollowingInitialDelayMs();
    long maxSleepTime = conf.getBlockWriteLocateFollowingMaxDelayMs();
    boolean fileComplete = false;
    int retries = conf.getNumBlockWriteLocateFollowingRetry();
    while (!fileComplete) {
      fileComplete =
          dfsClient.namenode.complete(src, dfsClient.clientName, last, fileId);
      if (!fileComplete) {
        final int hdfsTimeout = conf.getHdfsTimeout();
        if (!dfsClient.clientRunning
            || (hdfsTimeout > 0
                && localstart + hdfsTimeout < Time.monotonicNow())) {
          String msg = "Unable to close file because dfsclient " +
              " was unable to contact the HDFS servers. clientRunning " +
              dfsClient.clientRunning + " hdfsTimeout " + hdfsTimeout;
          DFSClient.LOG.info(msg);
          throw new IOException(msg);
        }
        try (TraceScope scope = dfsClient.getTracer()
            .newScope("DFSOutputStream#completeFile: Retry")) {
          scope.addKVAnnotation("retries left", retries);
          scope.addKVAnnotation("sleeptime (sleeping for)", sleeptime);
          if (retries == 0) {
            throw new IOException("Unable to close file because the last block "
                + last + " does not have enough number of replicas.");
          }
          retries--;
          Thread.sleep(sleeptime);
          sleeptime = calculateDelayForNextRetry(sleeptime, maxSleepTime);
          if (Time.monotonicNow() - localstart > 5000) {
            DFSClient.LOG.info("Could not complete " + src + " retrying...");
          }
        } catch (InterruptedException ie) {
          DFSClient.LOG.warn("Caught exception ", ie);
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Restore NN connectivity (wait out failover), then retry close() on the same output stream if you still hold it, or open-and-close is NOT the way - instead let the lease heal: the NameNode will eventually recover the lease (soft limit 60s / hard limit 1h defaults) and complete the file.
2. For deterministic recovery, run 'hdfs debug recoverLease -path <file>' after connectivity returns, then verify with 'hdfs fsck <file> -files -blocks'.
3. Call hflush() before close() in the write path so that even a failed close loses no acknowledged data - the file just waits for lease recovery.
4. Never tear down the FileSystem/DFSClient (FileSystem.close(), shutdown hooks) while other threads may still be closing output streams; sequence your shutdown.
5. If hdfsTimeout is set very low in your client config, raise it or set 0 (disabled) for long-finalizing files.

try {
  out.close();
} catch (IOException e) {
  if (!String.valueOf(e.getMessage()).contains("unable to contact the HDFS servers")) throw e;
  waitForNameNode(fs, 60_000); // poll fs.getFileStatus on a known path until NN answers
  // data already flushed is durable; finalize via lease recovery instead of retrying close:
  ((DistributedFileSystem) fs).recoverLease(path); // or: hdfs debug recoverLease -path
  verifyComplete(path); // hdfs fsck / getFileStatus shows the file closed
}

When it happens

Trigger: close() while the NameNode is unreachable or returning incomplete (e.g. NN failover in progress) AND the client is being shut down or the configured hdfs timeout expires; RPC layer retrying against a dead NN until the client gives up; DFSClient.close() invoked by a shutdown hook while a background thread is still closing files.

Common situations: Network partition or NN restart at the exact moment a job finalizes its output files; applications calling FileSystem.close() (clearing the client cache) while other threads still close their streams; JVM shutdown during close; clients with an aggressive hdfs timeout against a slow NN.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Got incremental block report from unregistered or dead node
• Block pool {bpid} has not recognized an active NN
• File is deleted: {} (inode {}) {}
• Client (={}) is not the lease owner (={}: {} (inode {}) {}
• The file {} belonged to does not exist or it is not under co

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7fcc07c3db88a87c. Report an issue: GitHub.