{"record":{"id":"c1261c4e7f61f983","repo":"apache/hadoop","slug":"unable-to-close-file-because-the-last-block-doe","errorCode":null,"errorMessage":"Unable to close file because the last block {} does not have enough number of replicas.","messagePattern":"Unable to close file because the last block (.+?) does not have enough number of replicas\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":1014,"sourceCode":"      fileComplete =\n          dfsClient.namenode.complete(src, dfsClient.clientName, last, fileId);\n      if (!fileComplete) {\n        final int hdfsTimeout = conf.getHdfsTimeout();\n        if (!dfsClient.clientRunning\n            || (hdfsTimeout > 0\n                && localstart + hdfsTimeout < Time.monotonicNow())) {\n          String msg = \"Unable to close file because dfsclient \" +\n              \" was unable to contact the HDFS servers. clientRunning \" +\n              dfsClient.clientRunning + \" hdfsTimeout \" + hdfsTimeout;\n          DFSClient.LOG.info(msg);\n          throw new IOException(msg);\n        }\n        try (TraceScope scope = dfsClient.getTracer()\n            .newScope(\"DFSOutputStream#completeFile: Retry\")) {\n          scope.addKVAnnotation(\"retries left\", retries);\n          scope.addKVAnnotation(\"sleeptime (sleeping for)\", sleeptime);\n          if (retries == 0) {\n            throw new IOException(\"Unable to close file because the last block \"\n                + last + \" does not have enough number of replicas.\");\n          }\n          retries--;\n          Thread.sleep(sleeptime);\n          sleeptime = calculateDelayForNextRetry(sleeptime, maxSleepTime);\n          if (Time.monotonicNow() - localstart > 5000) {\n            DFSClient.LOG.info(\"Could not complete \" + src + \" retrying...\");\n          }\n        } catch (InterruptedException ie) {\n          DFSClient.LOG.warn(\"Caught exception \", ie);\n        }\n      }\n    }\n  }\n\n  @VisibleForTesting\n  public void setArtificialSlowdown(long period) {\n    getStreamer().setArtificialSlowdown(period);","sourceCodeStart":996,"sourceCodeEnd":1032,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L996-L1032","documentation":"completeFile() gives up when the NameNode keeps answering 'not complete' for the final namenode.complete() call because the last block has not reached minimal replication (e.g. the pipeline lost DNs and the block is under-replicated). The client retries with exponential backoff (initial locateFollowingBlock delay, capped) up to dfs.client.block.write.locateFollowingBlock.retries (default 5) and then throws this IOException. The file stays in the NN's filesystem open-for-write under the client's lease until the lease is recovered or the block re-replicates and a later complete succeeds.","triggerScenarios":"close() when one or more DNs in the last block's pipeline died near the end of the write, so the NN cannot count the block as minimally replicated; replication=1 test files whose single DN is slow to block-report or was restarted; appends to files whose last block was already under-replicated before the append.","commonSituations":"Small/test clusters with dfs.replication=1 and DN restarts; clusters losing DNs (disk/network failures) during large batch writes; long-running writers that finish while the NN still shows stale block reports; the classic job symptom: task fails on close with this message after DN flakiness.","solutions":["Check and fix DN health (hdfs dfsadmin -report) and wait for the NN to re-replicate the under-replicated block (watch 'hdfs fsck / -list-underreplicated' empty out), then retry close() - or simply let lease recovery finalize the file (soft limit 60s if this client dies, hard limit 1h otherwise).","If you need the file closed now and can afford the current replica count, run 'hdfs debug recoverLease -path <file>' to force finalization, then verify data with 'hdfs fsck <file> -files -blocks'.","Raise dfs.client.block.write.locateFollowingBlock.retries on the client so close() tolerates longer re-replication windows.","For test clusters, keep dfs.replication >= 2 so a single DN restart cannot strand the last block."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  out.close();\n} catch (IOException e) {\n  if (!String.valueOf(e.getMessage()).contains(\"does not have enough number of replicas\")) throw e;\n  for (int i = 1; i <= 10; i++) {\n    Thread.sleep(10_000L);\n    if (underReplicatedBlocks(fs, path) == 0) break; // NN finished re-replication\n  }\n  ((DistributedFileSystem) fs).recoverLease(path); // force completion at current replication\n  verifyComplete(path);\n}","preventionTips":["Keep dfs.replication >= 2 so one lost DN cannot strand the last block.","Watch 'hdfs fsck -list-underreplicated' during DN flakiness and let it drain before batch close.","Raise dfs.client.block.write.locateFollowingBlock.retries for close() tolerance.","hflush() before close so data is safe even if completion is deferred to lease recovery."],"tags":["hdfs","hdfs-client","close","replication","datanode","under-replicated"],"backgroundTag":"under-replicated-block-on-close","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}