apache/hadoop · warning · InterruptedIOException

Interrupted while listing using DFS, prefix={}, marker={}

Error message

Interrupted while listing using DFS, prefix={}, marker={}

What it means

OBSFsDFSListing fans out per-level OBS list requests as Futures and joins them with Future.get(); an InterruptedException during the join logs 'Interrupted while listing using DFS, prefix=..., marker=...' and rethrows as InterruptedIOException with the same text. It signals the listing thread was interrupted (job kill, task cancellation, shutdown of the listing executor), not an OBS service failure. The sibling catch of ExecutionException handles actual worker failures separately.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSFsDFSListing.java:255

    }
    return newResultNum;
  }

  static void waitForOneLevelListTasksFinished(
      final List<ListObjectsRequest> oneLevelListRequests,
      final List<Future<ObjectListing>> oneLevelListFutures,
      final List<ObjectListing> oneLevelObjectListings)
      throws IOException {
    for (int i = 0; i < oneLevelListFutures.size(); i++) {
      try {
        oneLevelObjectListings.add(oneLevelListFutures.get(i).get());
      } catch (InterruptedException e) {
        LOG.warn("Interrupted while listing using DFS, prefix="
            + oneLevelListRequests.get(i).getPrefix() + ", marker="
            + (oneLevelListRequests.get(i).getMarker() != null
            ? oneLevelListRequests.get(i).getMarker()
            : ""));
        throw new InterruptedIOException(
            "Interrupted while listing using DFS, prefix="
                + oneLevelListRequests.get(i).getPrefix() + ", marker="
                + (oneLevelListRequests.get(i).getMarker() != null
                ? oneLevelListRequests.get(i).getMarker()
                : ""));
      } catch (ExecutionException e) {
        LOG.error("Exception while listing using DFS, prefix="
                + oneLevelListRequests.get(i).getPrefix() + ", marker="
                + (oneLevelListRequests.get(i).getMarker() != null
                ? oneLevelListRequests.get(i).getMarker()
                : ""),
            e);
        for (Future<ObjectListing> future : oneLevelListFutures) {
          future.cancel(true);
        }

        throw OBSCommonUtils.extractException(
            "Listing using DFS with exception, marker="

View on GitHub (pinned to 2add963021)

Solutions

1. If the interrupt is expected (cancellation), treat InterruptedIOException as a clean stop: restore the interrupt flag (Thread.currentThread().interrupt()) and exit the listing loop
2. For timeout-driven interrupts, switch to bounded listing (listStatusIterator / paginated listing with markers) rather than interrupting a blocking join
3. Avoid sharing the listing thread with code that calls interrupt() during shutdown; close the FileSystem instead
4. Re-run the job without cancellation if the interrupt was accidental (e.g. over-aggressive watchdog)

Example fix

// before
Thread t = new Thread(() -> fs.listStatus(deepDir));
t.interrupt(); // mid-listing -> InterruptedIOException: Interrupted while listing using DFS

// after
// cancellation-aware consumer
try {
  files = fs.listStatus(deepDir);
} catch (InterruptedIOException e) {
  Thread.currentThread().interrupt();
  return CANCELLED; // treat as clean cancellation, do not retry
}

try {
  return fs.listStatus(dir);
} catch (InterruptedIOException e) {
  if (String.valueOf(e.getMessage()).contains("listing using DFS")) {
    Thread.currentThread().interrupt(); // restore interrupt status
    return CANCELLED_SENTINEL; // treat as clean cancellation, no retry
  }
  throw e;
}

When it happens

Trigger: Directory-tree listing (listStatus on deep trees / whole-bucket traversal) in a task that gets cancelled — YARN preemption, Spark job cancellation, thread interrupt from a timeout wrapper; calling Thread.interrupt() on a thread blocked joining list results; JVM shutdown hooks interrupting in-flight listings.

Common situations: Interactive tools letting users cancel long listings; spark/yarn killing speculative or slow tasks mid-list; test harnesses that time out listings by interrupting the thread; CLI tools interrupted with Ctrl+C where the interrupt propagates into the listing thread.

Related errors

• Call interrupted
• Interrupted while waiting for IO on channel {}. Total timeou
• Read request interrupted
• Null IO stream
• Listing '%s' failed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/998dc27ae20d99e4. Report an issue: GitHub.