apache/hadoop · error · ReconfigurationException

Could not change property {property} from '{oldVal}' to '{ne

Error message

Could not change property {property} from '{oldVal}' to '{newVal}'

What it means

After a live reconfiguration of the datanode data directories (the volume case in reconfigurePropertyImpl), the DN refreshes volumes and then triggers a full block report so the NameNode acknowledges volume changes. If that block report (or an earlier step that set rootException) throws IOException, everything is wrapped in ReconfigurationException with the message 'Could not change property ... from ... to ...' — the new value is rejected from the operator's perspective and the underlying cause is chained.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:664

        LOG.info("Reconfiguring {} to {}", property, newVal);
        this.refreshVolumes(newVal);
        return getConf().get(DFS_DATANODE_DATA_DIR_KEY);
      } catch (IOException e) {
        rootException = e;
      } finally {
        // Send a full block report to let NN acknowledge the volume changes.
        try {
          triggerBlockReport(
              new BlockReportOptions.Factory().setIncremental(false).build());
        } catch (IOException e) {
          LOG.warn("Exception while sending the block report after refreshing"
              + " volumes {} to {}", property, newVal, e);
          if (rootException == null) {
            rootException = e;
          }
        } finally {
          if (rootException != null) {
            throw new ReconfigurationException(property, newVal,
                getConf().get(property), rootException);
          }
        }
      }
      break;
    }
    case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY: {
      ReconfigurationException rootException = null;
      try {
        LOG.info("Reconfiguring {} to {}", property, newVal);
        int movers;
        if (newVal == null) {
          // set to default
          movers = DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT;
        } else {
          movers = Integer.parseInt(newVal);
          if (movers <= 0) {
            rootException = new ReconfigurationException(

View on GitHub (pinned to 2add963021)

Solutions

Read the chained cause in the ReconfigurationException (and the 'Exception while sending the block report after refreshing volumes' WARN) — fix that first (NN reachability, directory permissions/paths).
Re-run the reconfiguration once the NameNode is stable; the previous failed attempt does not commit the property change.
Verify each new/changed directory exists, is writable by the DN user, and is mounted before reissuing.
Confirm success afterwards with -reconfig status and the DN's 'RECONFIGURE* changed' log line.

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: NN reachable and every target directory writable before reconfig
for (String dir : newDirList.split(",")) {
  Path p = new Path(dir);
  if (!p.getFileSystem(conf).isDirectory(p) || !isWritable(p)) {
    throw new IllegalStateException("Directory not ready: " + dir);
  }
}
assertNameNodeReachable(conf);

Type guard

static boolean isReconfigFailure(Exception e) {
  return e instanceof ReconfigurationException;
}

Try / catch

try {
  datanode.startReconfiguration(); // volume property change
} catch (ReconfigurationException e) {
  IOException cause = (IOException) e.getCause();
  LOG.warn("Volume reconfig failed ({}); retry after NN/dirs healthy",
      cause.getMessage());
  scheduleRetryAfterNameNodeIsUp(); // property change was not committed
}

Prevention

Run volume reconfiguration only while the NameNode is up and reachable (check with hdfs haadmin/dfsadmin first).
Pre-create and chmod new data directories before adding them live.
After any -reconfig run, verify with '-reconfig status' that the change actually applied.

When it happens

Trigger: 'hdfs dfsadmin -reconfig datanode ... -c' (or startReconfiguration) changing a volume property while the NN is unreachable or rejects the resulting full block report: NN down/failover in progress, RPC timeout, or volume refresh partially failed (a failed volume also sets rootException earlier in this case block).

Common situations: Hot-swapping disks during NN failover; reconfiguring dfs.datanode.dir on a DN whose NN connection is flaky; one new directory unreachable/unwritable so refreshVolumes recorded a failure.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b6494f2c84507414. Report an issue: GitHub.