apache/hadoop · error · UnsupportedOperationException

{} does not support listCorruptFileBlocks

Error message

{} does not support listCorruptFileBlocks

What it means

FileSystem.listCorruptFileBlocks(Path) is an optional API: the base implementation in FileSystem.java throws UnsupportedOperationException naming the concrete class. Only HDFS-backed filesystems implement it (Hdfs/DistributedFileSystem ask the NameNode for files whose blocks were found corrupt by DataNode block scanners; FilterFileSystem/FilterFs merely forward to the wrapped FS). The message deliberately names the class so you can see exactly which implementation lacks the capability.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:2099

      if (filter.accept(listing[i].getPath())) {
        results.add(listing[i]);
      }
    }
  }

  /**
   * List corrupted file blocks.
   *
   * @param path the path.
   * @return an iterator over the corrupt files under the given path
   * (may contain duplicates if a file has more than one corrupt block)
   * @throws UnsupportedOperationException if the operation is unsupported
   *         (default).
   * @throws IOException IO failure
   */
  public RemoteIterator<Path> listCorruptFileBlocks(Path path)
    throws IOException {
    throw new UnsupportedOperationException(getClass().getCanonicalName() +
        " does not support listCorruptFileBlocks");
  }

  /**
   * Filter files/directories in the given path using the user-supplied path
   * filter.
   * <p>
   * Does not guarantee to return the List of files/directories status in a
   * sorted order.
   *
   * @param f
   *          a path name
   * @param filter
   *          the user-supplied path filter
   * @return an array of FileStatus objects for the files under the given path
   *         after applying the filter
   * @throws FileNotFoundException when the path does not exist
   * @throws IOException see specific implementation

View on GitHub (pinned to 2add963021)

Solutions

  1. Run the call only on an HDFS-backed client: guard with fs instanceof DistributedFileSystem
  2. Probe the capability first: fs.hasPathCapability(path, CommonPathCapabilities.FS_LIST_CORRUPT_FILE_BLOCKS) (Hadoop 3.3.1+; default probe returns false rather than throwing)
  3. Catch UnsupportedOperationException and degrade gracefully (skip corrupt-block reporting for this store) — never retry it
  4. For ad-hoc checks use 'hdfs fsck <path> -list-corruptfileblocks' against the HDFS cluster instead of the client API

Example fix

// before
RemoteIterator<Path> corrupt = fs.listCorruptFileBlocks(dir);
// throws UnsupportedOperationException on file://, s3a://, har://

// after
if (fs.hasPathCapability(dir,
        CommonPathCapabilities.FS_LIST_CORRUPT_FILE_BLOCKS)) {
  RemoteIterator<Path> corrupt = fs.listCorruptFileBlocks(dir);
} else {
  LOG.warn("listCorruptFileBlocks unsupported on {}", fs.getUri());
}
Defensive patterns

Strategy: try-catch

Validate before calling

import org.apache.hadoop.fs.CommonPathCapabilities;

if (fs.hasPathCapability(dir,
        CommonPathCapabilities.FS_LIST_CORRUPT_FILE_BLOCKS)) {
  RemoteIterator<Path> it = fs.listCorruptFileBlocks(dir);
  // safe to consume
}

Type guard

static boolean supportsCorruptBlockListing(FileSystem fs) {
  return fs instanceof DistributedFileSystem;
}

Try / catch

try {
  RemoteIterator<Path> it = fs.listCorruptFileBlocks(dir);
} catch (UnsupportedOperationException e) {
  // capability is absent on this implementation: skip, never retry
  LOG.warn("listCorruptFileBlocks unsupported on {}: {}", fs.getUri(), e.getMessage());
}

Prevention

When it happens

Trigger: Calling fs.listCorruptFileBlocks(path) on any implementation that does not override it: LocalFileSystem/RawLocalFileSystem (file://), HarFileSystem, S3A, ABFS, GCS connectors, or a ViewFileSystem mount over a non-HDFS store. Also reached via FileContext.listCorruptFileBlocks or AbstractFileSystem defaults on such stores.

Common situations: Monitoring/fsck-like tooling written against hdfs:// is repointed at file:// or s3a:// via fs.defaultFS during unit tests or a migration; an ops script enumerates corrupt blocks against the wrong scheme; wrapper filesystems (checksum, view) that delegate to an unsupported inner FS.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1488a676174adda7. Report an issue: GitHub.