apache/hadoop · error · UnsupportedOperationException

Cannot getLocatedBlocks through a symlink to a non-Distribut

Error message

Cannot getLocatedBlocks through a symlink to a non-DistributedFileSystem: {} -> {}

What it means

DistributedFileSystem.getLocatedBlocks resolves the path through FileSystemLinkResolver; when the final hop of a symlink lands in a FileSystem that is not a DistributedFileSystem (local, s3a, viewfs target, etc.), it cannot fetch HDFS LocatedBlocks and throws UnsupportedOperationException. It is an inherent capability boundary: block-location queries only make sense inside HDFS. The message prints the target filesystem and path.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:4112

   * @return a LocatedBlocks object
   * @throws IOException
   */
  public LocatedBlocks getLocatedBlocks(Path p, long start, long len)
      throws IOException {
    final Path absF = fixRelativePart(p);
    return new FileSystemLinkResolver<LocatedBlocks>() {
      @Override
      public LocatedBlocks doCall(final Path p) throws IOException {
        return dfs.getLocatedBlocks(getPathName(p), start, len);
      }
      @Override
      public LocatedBlocks next(final FileSystem fs, final Path p)
          throws IOException {
        if (fs instanceof DistributedFileSystem) {
          DistributedFileSystem myDfs = (DistributedFileSystem)fs;
          return myDfs.getLocatedBlocks(p, start, len);
        }
        throw new UnsupportedOperationException("Cannot getLocatedBlocks " +
            "through a symlink to a non-DistributedFileSystem: " + fs + " -> "+
            p);
      }
    }.resolve(this, absF);
  }

  /**
   * Return path of the enclosing root for a given path
   * The enclosing root path is a common ancestor that should be used for temp and staging dirs
   * as well as within encryption zones and other restricted directories.
   *
   * @param path file path to find the enclosing root path for
   * @return a path to the enclosing root
   * @throws IOException early checks like failure to resolve path cause IO failures
   */
  public Path getEnclosingRoot(final Path path) throws IOException {
    statistics.incrementReadOps(1);
    storageStatistics.incrementOpCounter(OpType.GET_ENCLOSING_ROOT);

View on GitHub (pinned to 2add963021)

Solutions

Resolve the symlink first (FileContext.getFileStatus or FileStatus.getSymlink traversal) and only call block-location APIs when the resolved FS is a DistributedFileSystem
Fix or remove the HDFS symlink so it points to a real HDFS path if block-level operations are required
Use the target filesystem's own API for non-HDFS targets instead of HDFS block-location calls

Example fix

// before
LocatedBlocks lbs = ((DistributedFileSystem) fs)
    .getLocatedBlocks(symlinkPath, 0, Long.MAX_VALUE); // UnsupportedOperationException if target is not HDFS

// after
Path real = symlinkPath; // resolve manually if needed
FileSystem targetFs = real.getFileSystem(conf);
if (targetFs instanceof DistributedFileSystem) {
  LocatedBlocks lbs = ((DistributedFileSystem) targetFs)
      .getLocatedBlocks(real, 0, Long.MAX_VALUE);
} else {
  // handle non-HDFS target without block locations
}

Defensive patterns

Strategy: type-guard

Validate before calling

Path target = path;
FileSystem targetFs = FileSystem.get(target.toUri(), conf);
// follow one symlink hop if needed
if (fs.getFileLinkStatus(path).isSymlink()) {
  target = fs.getFileLinkStatus(path).getSymlink();
  targetFs = FileSystem.get(target.toUri(), conf);
}
boolean isHdfs = targetFs instanceof DistributedFileSystem;

Type guard

private static boolean resolvesToHdfs(FileSystem fs, Path p) throws IOException {
  FileSystem target = fs.getFileLinkStatus(p).isSymlink()
      ? FileSystem.get(fs.getFileLinkStatus(p).getSymlink().toUri(), fs.getConf())
      : fs;
  return target instanceof org.apache.hadoop.hdfs.DistributedFileSystem;
}

Try / catch

try {
  return dfs.getLocatedBlocks(p, start, len);
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("non-DistributedFileSystem")) {
    // degrade gracefully: no block locations for non-HDFS symlink targets
    return Collections.emptyList();
  }
  throw e;
}

Prevention

Before block-location or checksum calls, resolve symlinks and verify the target filesystem is DistributedFileSystem
Keep symlinks inside HDFS if tooling needs block-level metadata
Teach tools to skip or special-case non-HDFS symlink targets instead of assuming uniform HDFS layout

When it happens

Trigger: Calling an API that needs block locations (getFileBlockLocations, or internal getLocatedBlocks used by checksums/tools) on an HDFS path that is a symlink whose target resolves against another FileSystem scheme; symlink chains that leave HDFS via ViewFileSystem mounts or har:// / local targets.

Common situations: A user-created HDFS symlink pointing at a mount table entry or non-HDFS URI; data-lake layouts mixing hdfs:// and object-store mounts behind symlinks; tools (Spark locality, DistCp, checksum utilities) following symlinks across filesystem boundaries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/460be7963c9ca571. Report an issue: GitHub.