apache/hadoop · error · IOException

Rename of {} to {} is not allowed. The number of remote loca

Error message

Rename of {} to {} is not allowed. The number of remote locations for both source and target should be same.

What it means

In RouterClientProtocol.rename(), for a multi-destination directory (isMultiDestDirectory(src) true), the Router proxies rename to every remote location in parallel. getRenameDestinations() may trim locations whose destination-side mapping is absent; if the remaining list size differs from srcLocations.size(), the source/target location counts no longer line up and the Router aborts with IOException rather than partially renaming only some namespaces.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:638

      throws IOException {
    rpcServer.checkOperation(NameNode.OperationCategory.WRITE);

    final List<RemoteLocation> srcLocations =
        rpcServer.getLocationsForPath(src, true, false);
    final List<RemoteLocation> dstLocations =
        rpcServer.getLocationsForPath(dst, false, false);
    // srcLocations may be trimmed by getRenameDestinations()
    final List<RemoteLocation> locs = new LinkedList<>(srcLocations);
    RemoteParam dstParam = getRenameDestinations(locs, dstLocations);
    if (locs.isEmpty()) {
      return rbfRename.routerFedRename(src, dst, srcLocations, dstLocations);
    }
    RemoteMethod method = new RemoteMethod("rename",
        new Class<?>[] {String.class, String.class},
        new RemoteParam(), dstParam);
    if (isMultiDestDirectory(src)) {
      if (locs.size() != srcLocations.size()) {
        throw new IOException("Rename of " + src + " to " + dst + " is not"
            + " allowed. The number of remote locations for both source and"
            + " target should be same.");
      }
      return rpcClient.invokeAll(locs, method);
    } else {
      return rpcClient.invokeSequential(locs, method, Boolean.class,
          Boolean.TRUE);
    }
  }

  @Override
  public void rename2(final String src, final String dst,
      final Options.Rename... options) throws IOException {
    rpcServer.checkOperation(NameNode.OperationCategory.WRITE);

    final List<RemoteLocation> srcLocations =
        rpcServer.getLocationsForPath(src, true, false);
    final List<RemoteLocation> dstLocations =

View on GitHub (pinned to 2add963021)

Solutions

Align the destination counts: give dst the same set of nameservice destinations as src (update the mount entries) before renaming
Rename to a target inside the same multi-destination mount tree instead of across differently-shaped mounts
Perform the rename per-nameservice directly (hdfs dfs -fs hdfs://nsX) if federation-wide atomicity is not required
Check 'hdfs dsadmin -listMountTable' for src/dst mount entries and fix asymmetries

Example fix

# before: src has ns0,ns1 destinations; dst only ns0 -> rejected
hdfs dfs -fs hdfs://router -mv /multi/src /single/dst

# after: make dst a multi-dest mount with the same ns set
hdfs dsadmin -updateMount /single/dst ns0,ns1 /dst,/dst
hdfs dfs -fs hdfs://router -mv /multi/src /single/dst

Defensive patterns

Strategy: validation

Validate before calling

// Before renaming multi-destination mounts, confirm symmetric destination sets
List<RemoteLocation> srcLocs = rpcServer.getLocationsForPath(src, false, false);
List<RemoteLocation> dstLocs = rpcServer.getLocationsForPath(dst, false, false);
Set<String> srcNs = srcLocs.stream().map(RemoteLocation::getNameserviceId).collect(Collectors.toSet());
Set<String> dstNs = dstLocs.stream().map(RemoteLocation::getNameserviceId).collect(Collectors.toSet());
if (!srcNs.equals(dstNs)) {
  throw new IOException("Refusing rename: src/dst mount destinations differ: " + srcNs + " vs " + dstNs);
}

Try / catch

try {
  clientProtocol.rename(src, dst);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("number of remote locations")) {
    // align mount entries (same ns set for src and dst) or rename per-nameservice directly
  } else { throw e; }
}

Prevention

Design mount tables so multi-destination mounts have identical ns sets
Audit mount entries for asymmetry before federation-wide rename jobs
Prefer renames within a single mount subtree

When it happens

Trigger: rename()/rename2 via Router where src is a mount with multiple destinations and the dst mount has a different number of destinations or non-matching orderings (e.g. src mounted in ns0+ns1, dst mounted only in ns0, or dst default location ordering differs); getRenameDestinations drops a location because dst cannot resolve in that nameservice.

Common situations: Asymmetric mount tables (one side of the rename has 2 locations, the other 1); renaming between a multi-destination mount and a single-destination mount; mount tables edited concurrently so src and dst snapshots disagree.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1e1ad4e0e788feca. Report an issue: GitHub.