{"record":{"id":"998dc27ae20d99e4","repo":"apache/hadoop","slug":"interrupted-while-listing-using-dfs-prefix-ma","errorCode":null,"errorMessage":"Interrupted while listing using DFS, prefix={}, marker={}","messagePattern":"Interrupted while listing using DFS, prefix=(.+?), marker=(.+?)","errorType":"exception","errorClass":"InterruptedIOException","httpStatus":null,"severity":"warning","filePath":"hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSFsDFSListing.java","lineNumber":255,"sourceCode":"    }\n    return newResultNum;\n  }\n\n  static void waitForOneLevelListTasksFinished(\n      final List<ListObjectsRequest> oneLevelListRequests,\n      final List<Future<ObjectListing>> oneLevelListFutures,\n      final List<ObjectListing> oneLevelObjectListings)\n      throws IOException {\n    for (int i = 0; i < oneLevelListFutures.size(); i++) {\n      try {\n        oneLevelObjectListings.add(oneLevelListFutures.get(i).get());\n      } catch (InterruptedException e) {\n        LOG.warn(\"Interrupted while listing using DFS, prefix=\"\n            + oneLevelListRequests.get(i).getPrefix() + \", marker=\"\n            + (oneLevelListRequests.get(i).getMarker() != null\n            ? oneLevelListRequests.get(i).getMarker()\n            : \"\"));\n        throw new InterruptedIOException(\n            \"Interrupted while listing using DFS, prefix=\"\n                + oneLevelListRequests.get(i).getPrefix() + \", marker=\"\n                + (oneLevelListRequests.get(i).getMarker() != null\n                ? oneLevelListRequests.get(i).getMarker()\n                : \"\"));\n      } catch (ExecutionException e) {\n        LOG.error(\"Exception while listing using DFS, prefix=\"\n                + oneLevelListRequests.get(i).getPrefix() + \", marker=\"\n                + (oneLevelListRequests.get(i).getMarker() != null\n                ? oneLevelListRequests.get(i).getMarker()\n                : \"\"),\n            e);\n        for (Future<ObjectListing> future : oneLevelListFutures) {\n          future.cancel(true);\n        }\n\n        throw OBSCommonUtils.extractException(\n            \"Listing using DFS with exception, marker=\"","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSFsDFSListing.java#L237-L273","documentation":"OBSFsDFSListing fans out per-level OBS list requests as Futures and joins them with Future.get(); an InterruptedException during the join logs 'Interrupted while listing using DFS, prefix=..., marker=...' and rethrows as InterruptedIOException with the same text. It signals the listing thread was interrupted (job kill, task cancellation, shutdown of the listing executor), not an OBS service failure. The sibling catch of ExecutionException handles actual worker failures separately.","triggerScenarios":"Directory-tree listing (listStatus on deep trees / whole-bucket traversal) in a task that gets cancelled — YARN preemption, Spark job cancellation, thread interrupt from a timeout wrapper; calling Thread.interrupt() on a thread blocked joining list results; JVM shutdown hooks interrupting in-flight listings.","commonSituations":"Interactive tools letting users cancel long listings; spark/yarn killing speculative or slow tasks mid-list; test harnesses that time out listings by interrupting the thread; CLI tools interrupted with Ctrl+C where the interrupt propagates into the listing thread.","solutions":["If the interrupt is expected (cancellation), treat InterruptedIOException as a clean stop: restore the interrupt flag (Thread.currentThread().interrupt()) and exit the listing loop","For timeout-driven interrupts, switch to bounded listing (listStatusIterator / paginated listing with markers) rather than interrupting a blocking join","Avoid sharing the listing thread with code that calls interrupt() during shutdown; close the FileSystem instead","Re-run the job without cancellation if the interrupt was accidental (e.g. over-aggressive watchdog)"],"exampleFix":"// before\nThread t = new Thread(() -> fs.listStatus(deepDir));\nt.interrupt(); // mid-listing -> InterruptedIOException: Interrupted while listing using DFS\n\n// after\n// cancellation-aware consumer\ntry {\n  files = fs.listStatus(deepDir);\n} catch (InterruptedIOException e) {\n  Thread.currentThread().interrupt();\n  return CANCELLED; // treat as clean cancellation, do not retry\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return fs.listStatus(dir);\n} catch (InterruptedIOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"listing using DFS\")) {\n    Thread.currentThread().interrupt(); // restore interrupt status\n    return CANCELLED_SENTINEL; // treat as clean cancellation, no retry\n  }\n  throw e;\n}","preventionTips":["Don't interrupt listing threads for timeouts — use paginated listing with markers instead","Restore the interrupt flag whenever catching InterruptedIOException","Sequence shutdown: stop readers, then close the FileSystem, never interrupt mid-listing"],"tags":["interruption","listing","concurrency","hadoop-obs"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}