{"record":{"id":"8c5f4fcd2dfacc63","repo":"apache/hadoop","slug":"block-move-timed-out","errorCode":null,"errorMessage":"Block move timed out","messagePattern":"Block move timed out","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java","lineNumber":492,"sourceCode":"    /** Check whether to continue waiting for response */\n    private boolean stopWaitingForResponse(long startTime) {\n      return source.isIterationOver() ||\n          (blockMoveTimeout > 0 &&\n          (Time.monotonicNow() - startTime > blockMoveTimeout));\n    }\n\n    /** Receive a reportedBlock copy response from the input stream */\n    private void receiveResponse(DataInputStream in) throws IOException {\n      long startTime = Time.monotonicNow();\n      BlockOpResponseProto response =\n          BlockOpResponseProto.parseFrom(vintPrefixed(in));\n      while (response.getStatus() == Status.IN_PROGRESS) {\n        // read intermediate responses\n        response = BlockOpResponseProto.parseFrom(vintPrefixed(in));\n        // Stop waiting for slow block moves. Even if it stops waiting,\n        // the actual move may continue.\n        if (stopWaitingForResponse(startTime)) {\n          throw new IOException(\"Block move timed out\");\n        }\n      }\n      String logInfo = \"reportedBlock move is failed\";\n      DataTransferProtoUtil.checkBlockOpStatus(response, logInfo, true);\n    }\n\n    /** reset the object */\n    private void reset() {\n      reportedBlock = null;\n      source = null;\n      proxySource = null;\n      target = null;\n    }\n  }\n\n  private static boolean prepareRetryAfterInvalidEncryptionKey(KeyManager km,\n      int retryCount) throws IOException {\n    if (retryCount > 1) {","sourceCodeStart":474,"sourceCodeEnd":510,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java#L474-L510","documentation":"Thrown from Dispatcher.receiveResponse while polling block-move responses from the proxy source. stopWaitingForResponse returns true when the source iteration is over or (blockMoveTimeout > 0 and) the wait has exceeded dfs.balancer.block-move.timeout milliseconds; the balancer then stops waiting and raises IOException('Block move timed out'). Importantly the datanode-side copy may still complete - only the balancer's wait is abandoned, so the block can still be counted moved later.","triggerScenarios":"dfs.balancer.block-move.timeout set to a positive value smaller than the real time a replaceBlock operation takes (slow disks, throttled bandwidth dfs.datanode.balance.bandwidthPerSec, huge blocks, or datanodes under load), or the iteration time expiring mid-wait.","commonSituations":"Operators adding a block-move timeout to stop the balancer 'hanging' on stalled datanodes; aggressive bandwidth limits making every large block exceed the timeout; shared clusters where target datanodes are busy.","solutions":["Increase dfs.balancer.block-move.timeout (ms), or set it back to 0 (default) to wait indefinitely, then rerun the balancer","Raise dfs.datanode.balance.bandwidthPerSec or lower dfs.datanode.balance.max.concurrent.moves so individual moves finish faster","Investigate chronically slow datanodes (disk health, network) named in the surrounding move log lines"],"exampleFix":"# before\n<property><name>dfs.balancer.block-move.timeout</name><value>30000</value></property>\n\n# after\n<property><name>dfs.balancer.block-move.timeout</name><value>600000</value></property>","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage().equals(\"Block move timed out\")) { LOG.warn(\"Move wait abandoned; datanode copy may still finish\"); return; }\n  throw e;\n}","preventionTips":["Keep dfs.balancer.block-move.timeout=0 (wait forever) unless you specifically need stall detection","If set, keep it larger than (largest block size / balance bandwidth) with headroom","Rerun the balancer after fixing slow datanodes - timed-out moves are re-planned"],"tags":["hdfs","balancer","dispatcher","timeout","block-move"],"backgroundTag":"operation-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}