{"record":{"id":"f4aa83b651e303fb","repo":"apache/hadoop","slug":"link-resolution-does-not-work-with-multiple-file-s","errorCode":null,"errorMessage":"Link resolution does not work with multiple file systems for listLocatedStatus(): ","messagePattern":"Link resolution does not work with multiple file systems for listLocatedStatus\\(\\): ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java","lineNumber":1324,"sourceCode":"      final PathFilter filter)\n      throws IOException {\n    Path absF = fixRelativePart(p);\n    return new FileSystemLinkResolver<RemoteIterator<LocatedFileStatus>>() {\n      @Override\n      public RemoteIterator<LocatedFileStatus> doCall(final Path p)\n          throws IOException {\n        return new DirListingIterator<>(p, filter, true);\n      }\n\n      @Override\n      public RemoteIterator<LocatedFileStatus> next(final FileSystem fs,\n          final Path p) throws IOException {\n        if (fs instanceof DistributedFileSystem) {\n          return ((DistributedFileSystem)fs).listLocatedStatus(p, filter);\n        }\n        // symlink resolution for this methos does not work cross file systems\n        // because it is a protected method.\n        throw new IOException(\"Link resolution does not work with multiple \" +\n            \"file systems for listLocatedStatus(): \" + p);\n      }\n    }.resolve(this, absF);\n  }\n\n\n  /**\n   * Returns a remote iterator so that followup calls are made on demand\n   * while consuming the entries. This reduces memory consumption during\n   * listing of a large directory.\n   *\n   * @param p target path\n   * @return remote iterator\n   */\n  @Override\n  public RemoteIterator<FileStatus> listStatusIterator(final Path p)\n      throws IOException {\n    Path absF = fixRelativePart(p);","sourceCodeStart":1306,"sourceCodeEnd":1342,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L1306-L1342","documentation":"listLocatedStatus(Path, PathFilter) resolves symlinks via FileSystemLinkResolver; if the symlink target lands on a different FileSystem, the resolver falls back to next(fs, p) on that foreign filesystem. listLocatedStatus() is a protected method of FileSystem, so it cannot be invoked on the target filesystem instance, and the client throws IOException instead of silently doing an unlocated listing.","triggerScenarios":"Calling fs.listLocatedStatus(path) (or listFiles(path, filter) which uses located status) where path is an HDFS symlink whose target resolves onto a non-HDFS filesystem (e.g., symlink to a file:// or s3a:// path mounted via a different FileSystem scheme).","commonSituations":"Porting jobs that use symlinks between local and HDFS data; mixed-scheme data lakes where HDFS symlinks point into object stores; running code that worked on plain paths after someone replaced a directory with a cross-scheme symlink.","solutions":["Remove the cross-filesystem symlink and reference the target filesystem directly (open a FileSystem for the target URI and call listLocatedStatus/files on it).","Resolve the link yourself with FileContext or getLinkTarget, then call the target filesystem's public listFiles()/listLocatedStatus() equivalent on the resolved path.","Restructure data layout so HDFS symlinks only point to paths on the same HDFS instance.","As a last resort use the non-located listStatus() on the resolved target, which works cross-filesystem via public APIs."],"exampleFix":"// before\nRemoteIterator<LocatedFileStatus> it = hdfs.listLocatedStatus(linkPath); // IOException if link crosses filesystems\n\n// after\nPath target = hdfs.getLinkTarget(linkPath);\nFileSystem targetFs = target.toUri().getScheme() != null\n    ? FileSystem.get(target.toUri(), conf) : FileSystem.getLocal(conf);\nRemoteIterator<LocatedFileStatus> it =\n    (RemoteIterator<LocatedFileStatus>) (RemoteIterator<?>) targetFs.listFiles(target, true);","handlingStrategy":"fallback","validationCode":"FileStatus st = hdfs.getFileLinkStatus(p);\nif (st.isSymlink()) {\n  Path target = st.getSymlink();\n  // cross-filesystem only if scheme/authority differs from hdfs.getUri()\n  boolean crossFs = !hdfs.getUri().equals(\n      new Path(hdfs.makeQualified(target), \"\").toUri());\n}","typeGuard":null,"tryCatchPattern":"try {\n  it = hdfs.listLocatedStatus(p);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"listLocatedStatus\")) {\n    // resolve target and list on its filesystem instead\n  } else throw e;\n}","preventionTips":["Do not create HDFS symlinks whose targets live on other schemes (file://, s3a://, etc.).","Centralize symlink policy in one utility that refuses cross-scheme links at creation time.","Document which listing APIs are protected (listLocatedStatus) and therefore break under cross-FS resolution."],"tags":["hdfs","symlink","cross-filesystem","listlocatedstatus","filesystemlinkresolver","distributed-filesystem"],"backgroundTag":"symlink-cross-filesystem-unsupported","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}