{"record":{"id":"7fcc07c3db88a87c","repo":"apache/hadoop","slug":"unable-to-close-file-because-dfsclient-was-unable","errorCode":null,"errorMessage":"Unable to close file because dfsclient  was unable to contact the HDFS servers. clientRunning {} hdfsTimeout {}","messagePattern":"Unable to close file because dfsclient  was unable to contact the HDFS servers\\. clientRunning (.+?) hdfsTimeout (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":1007,"sourceCode":"    long localstart = Time.monotonicNow();\n    final DfsClientConf conf = dfsClient.getConf();\n    long sleeptime = conf.getBlockWriteLocateFollowingInitialDelayMs();\n    long maxSleepTime = conf.getBlockWriteLocateFollowingMaxDelayMs();\n    boolean fileComplete = false;\n    int retries = conf.getNumBlockWriteLocateFollowingRetry();\n    while (!fileComplete) {\n      fileComplete =\n          dfsClient.namenode.complete(src, dfsClient.clientName, last, fileId);\n      if (!fileComplete) {\n        final int hdfsTimeout = conf.getHdfsTimeout();\n        if (!dfsClient.clientRunning\n            || (hdfsTimeout > 0\n                && localstart + hdfsTimeout < Time.monotonicNow())) {\n          String msg = \"Unable to close file because dfsclient \" +\n              \" was unable to contact the HDFS servers. clientRunning \" +\n              dfsClient.clientRunning + \" hdfsTimeout \" + hdfsTimeout;\n          DFSClient.LOG.info(msg);\n          throw new IOException(msg);\n        }\n        try (TraceScope scope = dfsClient.getTracer()\n            .newScope(\"DFSOutputStream#completeFile: Retry\")) {\n          scope.addKVAnnotation(\"retries left\", retries);\n          scope.addKVAnnotation(\"sleeptime (sleeping for)\", sleeptime);\n          if (retries == 0) {\n            throw new IOException(\"Unable to close file because the last block \"\n                + last + \" does not have enough number of replicas.\");\n          }\n          retries--;\n          Thread.sleep(sleeptime);\n          sleeptime = calculateDelayForNextRetry(sleeptime, maxSleepTime);\n          if (Time.monotonicNow() - localstart > 5000) {\n            DFSClient.LOG.info(\"Could not complete \" + src + \" retrying...\");\n          }\n        } catch (InterruptedException ie) {\n          DFSClient.LOG.warn(\"Caught exception \", ie);\n        }","sourceCodeStart":989,"sourceCodeEnd":1025,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L989-L1025","documentation":"completeFile() loops calling namenode.complete() until the NN confirms the file is done. Each 'false' answer means the NN could not yet complete the file, and the client bails out with this IOException when either dfsClient.clientRunning is false (the DFSClient is shutting down - e.g. FileSystem.close()/static cache clear, JVM shutdown hook, or an abort) or the client HDFS timeout (hdfsTimeout, 0 = disabled) elapsed since the close started. The message itself is logged at INFO before being thrown.","triggerScenarios":"close() while the NameNode is unreachable or returning incomplete (e.g. NN failover in progress) AND the client is being shut down or the configured hdfs timeout expires; RPC layer retrying against a dead NN until the client gives up; DFSClient.close() invoked by a shutdown hook while a background thread is still closing files.","commonSituations":"Network partition or NN restart at the exact moment a job finalizes its output files; applications calling FileSystem.close() (clearing the client cache) while other threads still close their streams; JVM shutdown during close; clients with an aggressive hdfs timeout against a slow NN.","solutions":["Restore NN connectivity (wait out failover), then retry close() on the same output stream if you still hold it, or open-and-close is NOT the way - instead let the lease heal: the NameNode will eventually recover the lease (soft limit 60s / hard limit 1h defaults) and complete the file.","For deterministic recovery, run 'hdfs debug recoverLease -path <file>' after connectivity returns, then verify with 'hdfs fsck <file> -files -blocks'.","Call hflush() before close() in the write path so that even a failed close loses no acknowledged data - the file just waits for lease recovery.","Never tear down the FileSystem/DFSClient (FileSystem.close(), shutdown hooks) while other threads may still be closing output streams; sequence your shutdown.","If hdfsTimeout is set very low in your client config, raise it or set 0 (disabled) for long-finalizing files."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  out.close();\n} catch (IOException e) {\n  if (!String.valueOf(e.getMessage()).contains(\"unable to contact the HDFS servers\")) throw e;\n  waitForNameNode(fs, 60_000); // poll fs.getFileStatus on a known path until NN answers\n  // data already flushed is durable; finalize via lease recovery instead of retrying close:\n  ((DistributedFileSystem) fs).recoverLease(path); // or: hdfs debug recoverLease -path\n  verifyComplete(path); // hdfs fsck / getFileStatus shows the file closed\n}","preventionTips":["hflush() periodically during the write so a failed close loses nothing.","Sequence process shutdown so FileSystem/DFSClient is not closed while streams still finalize.","Monitor NN HA failover and back off closes during one instead of timing out.","Know the lease path: soft-limit 60s after this client dies, hard-limit 1h otherwise - the file completes on its own."],"tags":["hdfs","hdfs-client","close","namenode","connectivity","lease","failover"],"backgroundTag":"namenode-unreachable-during-close","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}