{"record":{"id":"b6494f2c84507414","repo":"apache/hadoop","slug":"could-not-change-property-property-from-oldval","errorCode":null,"errorMessage":"Could not change property {property} from '{oldVal}' to '{newVal}'","messagePattern":"Could not change property (.+?) from '(.+?)' to '(.+?)'","errorType":"exception","errorClass":"ReconfigurationException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java","lineNumber":664,"sourceCode":"        LOG.info(\"Reconfiguring {} to {}\", property, newVal);\n        this.refreshVolumes(newVal);\n        return getConf().get(DFS_DATANODE_DATA_DIR_KEY);\n      } catch (IOException e) {\n        rootException = e;\n      } finally {\n        // Send a full block report to let NN acknowledge the volume changes.\n        try {\n          triggerBlockReport(\n              new BlockReportOptions.Factory().setIncremental(false).build());\n        } catch (IOException e) {\n          LOG.warn(\"Exception while sending the block report after refreshing\"\n              + \" volumes {} to {}\", property, newVal, e);\n          if (rootException == null) {\n            rootException = e;\n          }\n        } finally {\n          if (rootException != null) {\n            throw new ReconfigurationException(property, newVal,\n                getConf().get(property), rootException);\n          }\n        }\n      }\n      break;\n    }\n    case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY: {\n      ReconfigurationException rootException = null;\n      try {\n        LOG.info(\"Reconfiguring {} to {}\", property, newVal);\n        int movers;\n        if (newVal == null) {\n          // set to default\n          movers = DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT;\n        } else {\n          movers = Integer.parseInt(newVal);\n          if (movers <= 0) {\n            rootException = new ReconfigurationException(","sourceCodeStart":646,"sourceCodeEnd":682,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L646-L682","documentation":"After a live reconfiguration of the datanode data directories (the volume case in reconfigurePropertyImpl), the DN refreshes volumes and then triggers a full block report so the NameNode acknowledges volume changes. If that block report (or an earlier step that set rootException) throws IOException, everything is wrapped in ReconfigurationException with the message 'Could not change property ... from ... to ...' — the new value is rejected from the operator's perspective and the underlying cause is chained.","triggerScenarios":"'hdfs dfsadmin -reconfig datanode ... -c' (or startReconfiguration) changing a volume property while the NN is unreachable or rejects the resulting full block report: NN down/failover in progress, RPC timeout, or volume refresh partially failed (a failed volume also sets rootException earlier in this case block).","commonSituations":"Hot-swapping disks during NN failover; reconfiguring dfs.datanode.dir on a DN whose NN connection is flaky; one new directory unreachable/unwritable so refreshVolumes recorded a failure.","solutions":["Read the chained cause in the ReconfigurationException (and the 'Exception while sending the block report after refreshing volumes' WARN) — fix that first (NN reachability, directory permissions/paths).","Re-run the reconfiguration once the NameNode is stable; the previous failed attempt does not commit the property change.","Verify each new/changed directory exists, is writable by the DN user, and is mounted before reissuing.","Confirm success afterwards with -reconfig status and the DN's 'RECONFIGURE* changed' log line."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-flight: NN reachable and every target directory writable before reconfig\nfor (String dir : newDirList.split(\",\")) {\n  Path p = new Path(dir);\n  if (!p.getFileSystem(conf).isDirectory(p) || !isWritable(p)) {\n    throw new IllegalStateException(\"Directory not ready: \" + dir);\n  }\n}\nassertNameNodeReachable(conf);","typeGuard":"static boolean isReconfigFailure(Exception e) {\n  return e instanceof ReconfigurationException;\n}","tryCatchPattern":"try {\n  datanode.startReconfiguration(); // volume property change\n} catch (ReconfigurationException e) {\n  IOException cause = (IOException) e.getCause();\n  LOG.warn(\"Volume reconfig failed ({}); retry after NN/dirs healthy\",\n      cause.getMessage());\n  scheduleRetryAfterNameNodeIsUp(); // property change was not committed\n}","preventionTips":["Run volume reconfiguration only while the NameNode is up and reachable (check with hdfs haadmin/dfsadmin first).","Pre-create and chmod new data directories before adding them live.","After any -reconfig run, verify with '-reconfig status' that the change actually applied."],"tags":["hdfs","datanode","reconfiguration","volumes","block-report"],"backgroundTag":"reconfiguration-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}