apache/hadoop · error · IllegalArgumentException

File not found in downstream nameservices: {}

Error message

File not found in downstream nameservices: {}

What it means

When dfs.federation.router.admin.mount.check.enable is true (default false), RouterAdminServer.addMountTableEntry/addMountTableEntries/updateMountTableEntry call verifyFileExistenceInDest, which checks each mount destination via getFileInfo against the remote nameservices. Any nameservice where the destination path is absent is collected and reported as IllegalArgumentException('File not found in downstream nameservices: ns1,ns2'). This prevents dangling mount entries.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java:424

              != newQuota.getTypeQuota(t)) {
            synchronizeQuota(updateEntry.getSourcePath(),
                HdfsConstants.QUOTA_DONT_SET, newQuota.getTypeQuota(t), t);
          }
        }
      }
    } catch (Exception e) {
      // Ignore exception, if any while reseting quota. Specifically to handle
      // if the actual destination doesn't exist.
      LOG.warn("Unable to reset quota at the destinations for {}: {}",
          request.getEntry(), e.getMessage());
    }
    return response;
  }

  private void verifyFileExistenceInDest(MountTable mountTable) throws IOException {
    List<String> nsIds = verifyFileInDestinations(mountTable);
    if (!nsIds.isEmpty()) {
      throw new IllegalArgumentException(
          "File not found in downstream nameservices: " + StringUtils.join(",", nsIds));
    }
  }

  /**
   * Checks whether quota needs to be synchronized with namespace or not. Quota
   * needs to be synchronized either if there is change in mount entry quota or
   * there is change in remote destinations.
   * @param request the update request.
   * @param oldEntry the mount entry before getting updated.
   * @return true if quota needs to be updated.
   * @throws IOException
   */
  private boolean isQuotaUpdated(UpdateMountTableEntryRequest request,
      MountTable oldEntry) throws IOException {
    if (oldEntry != null) {
      MountTable updateEntry = request.getEntry();
      // If locations are changed, the new destinations need to be in sync with

View on GitHub (pinned to 2add963021)

Solutions

1. Create the destination path in every listed nameservice first (hdfs dfs -fs hdfs://<ns> -mkdir -p /dest), then retry the dsadmin command
2. Double-check the destination paths and nameservice IDs in the dsadmin arguments for typos
3. If pre-creation is undesired (e.g. lazy provisioning), set dfs.federation.router.admin.mount.check.enable=false and accept dangling mounts

Example fix

# before: fails when /dest missing in ns1
hdfs dsadmin -addMount /data ns1 /dest

# after: pre-create destination in each ns, then add
hdfs dfs -fs hdfs://ns1 -mkdir -p /dest
hdfs dsadmin -addMount /data ns1 /dest

// With mount.check.enable=true, pre-create destinations before dsadmin add/update
for (RemoteLocation dest : mountTable.getDestinations()) {
  FileSystem fs = FileSystem.get(new URI("hdfs://" + dest.getNameserviceId()), conf);
  if (!fs.exists(new Path(dest.getDest()))) {
    fs.mkdirs(new Path(dest.getDest()));
  }
}

try {
  routerAdmin.addMountTableEntry(req);
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("File not found in downstream nameservices")) {
    // parse listed nsIds, create the destination path there, retry the command
  } else { throw e; }
}

When it happens

Trigger: hdfs dsadmin -addMount /path ns1 /dest where /dest does not exist in ns1's NameNode, with mount.check.enable=true; updating a mount to new remote locations that were not pre-created; destination created in only some of the target nameservices (multi-destination mount).

Common situations: Enabling destination checking on an existing federation where older mounts were added without it; operators forgetting to hdfs dfs -mkdir the destination path in the subcluster before adding the mount; typos in the remote destination path.

Related errors

• Mount table state store is not available.
• Permission denied: {} is not allowed to change quota of {}
• Disabled Nameservice state store is not available.
• Rename of {} to {} is not allowed. The number of remote loca
• Cannot find target file - {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a64f33ae3286bf81. Report an issue: GitHub.