{"record":{"id":"a64f33ae3286bf81","repo":"apache/hadoop","slug":"file-not-found-in-downstream-nameservices","errorCode":null,"errorMessage":"File not found in downstream nameservices: {}","messagePattern":"File not found in downstream nameservices: (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java","lineNumber":424,"sourceCode":"              != newQuota.getTypeQuota(t)) {\n            synchronizeQuota(updateEntry.getSourcePath(),\n                HdfsConstants.QUOTA_DONT_SET, newQuota.getTypeQuota(t), t);\n          }\n        }\n      }\n    } catch (Exception e) {\n      // Ignore exception, if any while reseting quota. Specifically to handle\n      // if the actual destination doesn't exist.\n      LOG.warn(\"Unable to reset quota at the destinations for {}: {}\",\n          request.getEntry(), e.getMessage());\n    }\n    return response;\n  }\n\n  private void verifyFileExistenceInDest(MountTable mountTable) throws IOException {\n    List<String> nsIds = verifyFileInDestinations(mountTable);\n    if (!nsIds.isEmpty()) {\n      throw new IllegalArgumentException(\n          \"File not found in downstream nameservices: \" + StringUtils.join(\",\", nsIds));\n    }\n  }\n\n  /**\n   * Checks whether quota needs to be synchronized with namespace or not. Quota\n   * needs to be synchronized either if there is change in mount entry quota or\n   * there is change in remote destinations.\n   * @param request the update request.\n   * @param oldEntry the mount entry before getting updated.\n   * @return true if quota needs to be updated.\n   * @throws IOException\n   */\n  private boolean isQuotaUpdated(UpdateMountTableEntryRequest request,\n      MountTable oldEntry) throws IOException {\n    if (oldEntry != null) {\n      MountTable updateEntry = request.getEntry();\n      // If locations are changed, the new destinations need to be in sync with","sourceCodeStart":406,"sourceCodeEnd":442,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java#L406-L442","documentation":"When dfs.federation.router.admin.mount.check.enable is true (default false), RouterAdminServer.addMountTableEntry/addMountTableEntries/updateMountTableEntry call verifyFileExistenceInDest, which checks each mount destination via getFileInfo against the remote nameservices. Any nameservice where the destination path is absent is collected and reported as IllegalArgumentException('File not found in downstream nameservices: ns1,ns2'). This prevents dangling mount entries.","triggerScenarios":"hdfs dsadmin -addMount /path ns1 /dest where /dest does not exist in ns1's NameNode, with mount.check.enable=true; updating a mount to new remote locations that were not pre-created; destination created in only some of the target nameservices (multi-destination mount).","commonSituations":"Enabling destination checking on an existing federation where older mounts were added without it; operators forgetting to hdfs dfs -mkdir the destination path in the subcluster before adding the mount; typos in the remote destination path.","solutions":["Create the destination path in every listed nameservice first (hdfs dfs -fs hdfs://<ns> -mkdir -p /dest), then retry the dsadmin command","Double-check the destination paths and nameservice IDs in the dsadmin arguments for typos","If pre-creation is undesired (e.g. lazy provisioning), set dfs.federation.router.admin.mount.check.enable=false and accept dangling mounts"],"exampleFix":"# before: fails when /dest missing in ns1\nhdfs dsadmin -addMount /data ns1 /dest\n\n# after: pre-create destination in each ns, then add\nhdfs dfs -fs hdfs://ns1 -mkdir -p /dest\nhdfs dsadmin -addMount /data ns1 /dest","handlingStrategy":"validation","validationCode":"// With mount.check.enable=true, pre-create destinations before dsadmin add/update\nfor (RemoteLocation dest : mountTable.getDestinations()) {\n  FileSystem fs = FileSystem.get(new URI(\"hdfs://\" + dest.getNameserviceId()), conf);\n  if (!fs.exists(new Path(dest.getDest()))) {\n    fs.mkdirs(new Path(dest.getDest()));\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  routerAdmin.addMountTableEntry(req);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"File not found in downstream nameservices\")) {\n    // parse listed nsIds, create the destination path there, retry the command\n  } else { throw e; }\n}","preventionTips":["Automate destination-path creation as part of the mount-provisioning runbook","Keep mount.check.enable=true in prod to catch dangling destinations early","Validate destination paths and nsIds with a dry-run before applying"],"tags":["hdfs","router-based-federation","mount-table","admin","validation"],"backgroundTag":"path-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}