apache/hadoop · error · IOException

Link resolution does not work with multiple file systems for

Error message

Link resolution does not work with multiple file systems for listLocatedStatus():

What it means

listLocatedStatus(Path, PathFilter) resolves symlinks via FileSystemLinkResolver; if the symlink target lands on a different FileSystem, the resolver falls back to next(fs, p) on that foreign filesystem. listLocatedStatus() is a protected method of FileSystem, so it cannot be invoked on the target filesystem instance, and the client throws IOException instead of silently doing an unlocated listing.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:1324

      final PathFilter filter)
      throws IOException {
    Path absF = fixRelativePart(p);
    return new FileSystemLinkResolver<RemoteIterator<LocatedFileStatus>>() {
      @Override
      public RemoteIterator<LocatedFileStatus> doCall(final Path p)
          throws IOException {
        return new DirListingIterator<>(p, filter, true);
      }

      @Override
      public RemoteIterator<LocatedFileStatus> next(final FileSystem fs,
          final Path p) throws IOException {
        if (fs instanceof DistributedFileSystem) {
          return ((DistributedFileSystem)fs).listLocatedStatus(p, filter);
        }
        // symlink resolution for this methos does not work cross file systems
        // because it is a protected method.
        throw new IOException("Link resolution does not work with multiple " +
            "file systems for listLocatedStatus(): " + p);
      }
    }.resolve(this, absF);
  }


  /**
   * Returns a remote iterator so that followup calls are made on demand
   * while consuming the entries. This reduces memory consumption during
   * listing of a large directory.
   *
   * @param p target path
   * @return remote iterator
   */
  @Override
  public RemoteIterator<FileStatus> listStatusIterator(final Path p)
      throws IOException {
    Path absF = fixRelativePart(p);

View on GitHub (pinned to 2add963021)

Solutions

Remove the cross-filesystem symlink and reference the target filesystem directly (open a FileSystem for the target URI and call listLocatedStatus/files on it).
Resolve the link yourself with FileContext or getLinkTarget, then call the target filesystem's public listFiles()/listLocatedStatus() equivalent on the resolved path.
Restructure data layout so HDFS symlinks only point to paths on the same HDFS instance.
As a last resort use the non-located listStatus() on the resolved target, which works cross-filesystem via public APIs.

Example fix

// before
RemoteIterator<LocatedFileStatus> it = hdfs.listLocatedStatus(linkPath); // IOException if link crosses filesystems

// after
Path target = hdfs.getLinkTarget(linkPath);
FileSystem targetFs = target.toUri().getScheme() != null
    ? FileSystem.get(target.toUri(), conf) : FileSystem.getLocal(conf);
RemoteIterator<LocatedFileStatus> it =
    (RemoteIterator<LocatedFileStatus>) (RemoteIterator<?>) targetFs.listFiles(target, true);

Defensive patterns

Strategy: fallback

Validate before calling

FileStatus st = hdfs.getFileLinkStatus(p);
if (st.isSymlink()) {
  Path target = st.getSymlink();
  // cross-filesystem only if scheme/authority differs from hdfs.getUri()
  boolean crossFs = !hdfs.getUri().equals(
      new Path(hdfs.makeQualified(target), "").toUri());
}

Try / catch

try {
  it = hdfs.listLocatedStatus(p);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("listLocatedStatus")) {
    // resolve target and list on its filesystem instead
  } else throw e;
}

Prevention

Do not create HDFS symlinks whose targets live on other schemes (file://, s3a://, etc.).
Centralize symlink policy in one utility that refuses cross-scheme links at creation time.
Document which listing APIs are protected (listLocatedStatus) and therefore break under cross-FS resolution.

When it happens

Trigger: Calling fs.listLocatedStatus(path) (or listFiles(path, filter) which uses located status) where path is an HDFS symlink whose target resolves onto a non-HDFS filesystem (e.g., symlink to a file:// or s3a:// path mounted via a different FileSystem scheme).

Common situations: Porting jobs that use symlinks between local and HDFS data; mixed-scheme data lakes where HDFS symlinks point into object stores; running code that worked on plain paths after someone replaced a directory with a cross-scheme symlink.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f4aa83b651e303fb. Report an issue: GitHub.