{"record":{"id":"ef578898fa0745f6","repo":"apache/hadoop","slug":"block-move-cancelled","errorCode":null,"errorMessage":"Block move cancelled.","messagePattern":"Block move cancelled\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"info","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java","lineNumber":371,"sourceCode":"      final DDatanode dn = g.getDDatanode();\n      if (dn.addPendingBlock(this)) {\n        proxySource = dn;\n        return true;\n      }\n      return false;\n    }\n\n    /** Dispatch the move to the proxy source & wait for the response. */\n    private void dispatch() {\n      Socket sock = new Socket();\n      DataOutputStream out = null;\n      DataInputStream in = null;\n      try {\n        if (source.isIterationOver()){\n          LOG.info(\"Cancel moving \" + this +\n              \" as iteration is already cancelled due to\" +\n              \" dfs.balancer.max-iteration-time is passed.\");\n          throw new IOException(\"Block move cancelled.\");\n        }\n        LOG.info(\"Start moving \" + this);\n        assert !(reportedBlock instanceof DBlockStriped);\n\n        ExtendedBlock eb = new ExtendedBlock(nnc.getBlockpoolID(),\n            reportedBlock.getBlock());\n        final KeyManager km = nnc.getKeyManager();\n        Token<BlockTokenIdentifier> accessToken = null;\n        OutputStream unbufOut;\n        InputStream unbufIn;\n        int encryptionKeyRetryCount = 0;\n        while (true) {\n          try {\n            accessToken = km.getAccessToken(eb,\n                new StorageType[]{target.storageType}, new String[0]);\n            sock.connect(\n                NetUtils.createSocketAddr(target.getDatanodeInfo().\n                    getXferAddr(Dispatcher.this.connectToDnViaHostname)),","sourceCodeStart":353,"sourceCodeEnd":389,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java#L353-L389","documentation":"Raised inside Dispatcher.dispatch when a queued block move is about to be sent but its source StorageGroup is already flagged iteration-over - i.e. dfs.balancer.max-iteration-time elapsed and this iteration was cancelled. It is a controlled cancellation, not a fault: the move is skipped and counted as failed for this iteration, and the info log right above states the reason. The balancer exits normally afterwards.","triggerScenarios":"Setting dfs.balancer.max-iteration-time to a small value (or a genuinely long iteration) so that moves still in the queue when time expires hit source.isIterationOver() == true.","commonSituations":"Clusters that cap balancer iterations to bound impact during business hours; very large rebalances where the dispatch queue outlives the iteration window.","solutions":["Treat as expected if you set a short dfs.balancer.max-iteration-time; run the balancer again - remaining blocks are handled in the next run","Increase dfs.balancer.max-iteration-time (ms) so an iteration can drain its move queue","Reduce per-iteration work (lower dfs.datanode.balance.max.concurrent.moves or a smaller -threshold) so fewer moves are pending when time expires"],"exampleFix":"# before\n<property><name>dfs.balancer.max-iteration-time</name><value>300000</value></property>\n\n# after\n<property><name>dfs.balancer.max-iteration-time</name><value>1800000</value></property>","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage().contains(\"Block move cancelled\")) { LOG.debug(\"Iteration deadline hit; move will be retried next run\"); }\n  else { throw e; }\n}","preventionTips":["Size dfs.balancer.max-iteration-time generously relative to cluster size, or accept and reschedule follow-up runs","Monitor balancer summary lines for 'already cancelled' messages to tune the iteration window"],"tags":["hdfs","balancer","dispatcher","cancellation","lifecycle"],"backgroundTag":"operation-cancelled","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}