{"record":{"id":"bf9b53d13324c96f","repo":"apache/druid","slug":"could-not-close-channel-for-level-d-and-rank","errorCode":null,"errorMessage":"Could not close channel for level [%d] and rank [%d]","messagePattern":"Could not close channel for level \\[(.+?)\\] and rank \\[(.+?)\\]","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/frame/processor/SuperSorter.java","lineNumber":783,"sourceCode":"        synchronized (runWorkersLock) {\n          outputsReadyByLevel.computeIfAbsent(level, ignored2 -> new LongRBTreeSet())\n                             .add(rank);\n          superSorterProgressTracker.addMergedBatchesForLevel(level, 1);\n\n          if (isLimited() && totalMergingLevels != UNKNOWN_LEVEL && level == totalMergingLevels - 1) {\n            rowLimit -= outputRows;\n\n            if (rowLimit < 0) {\n              throw DruidException.defensive(\"rowLimit[%d] below zero after outputRows[%d]\", rowLimit, outputRows);\n            }\n          }\n\n          for (PartitionedReadableFrameChannel partitionedReadableFrameChannel : partitionedReadableChannelsToClose) {\n            try {\n              partitionedReadableFrameChannel.close();\n            }\n            catch (IOException e) {\n              throw new UncheckedIOException(\n                  StringUtils.format(\"Could not close channel for level [%d] and rank [%d]\", level, rank),\n                  e\n              );\n            }\n          }\n        }\n      });\n    }\n    catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n  }\n\n  private <T> void runWorker(final FrameProcessor<T> worker, final Consumer<T> outConsumer)\n  {\n    Futures.addCallback(\n        exec.runFully(processorDecorator.decorate(worker), cancellationId),\n        new FutureCallback<>()","sourceCodeStart":765,"sourceCodeEnd":801,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/frame/processor/SuperSorter.java#L765-L801","documentation":"When a merge step finishes, SuperSorter closes the PartitionedReadableFrameChannels it consumed. If closing one throws an IOException, it is wrapped in an UncheckedIOException identifying the merge level and rank. This indicates a resource cleanup failure — often a symptom of an underlying storage or filesystem problem rather than a bug in merge logic itself.","triggerScenarios":"An IOException thrown while closing a partitioned readable frame channel in runMerger's cleanup path — typically when the channel is backed by spilled frames on disk and close() triggers file/stream cleanup that fails (disk full during deletion bookkeeping, NFS/stale file handle, file already removed, permission problems).","commonSituations":"Task working directory cleaned up concurrently by another process while the merger closes channels; network filesystems (HDFS/NFS/S3-backed spill) failing on close; disk-full or I/O errors on the node running the task; leaked descriptors hitting OS limits causing close failures.","solutions":["Inspect the underlying IOException cause and the affected node's disk/filesystem health (dmesg, disk space).","Check that the task's spill/working directories are not being modified or deleted externally while the query runs.","Re-run the query/task; transient storage I/O errors often clear on retry.","If persistent, verify filesystem permissions and mount health for druid's temporary/spill directories."],"exampleFix":"// before (fragile cleanup location)\n// spill directory shared and deleted by external cleanup cron\n// after\n// exclude active task tmp dirs from external cleanup, or keep spill under druid's managed task dir\ncleanup_job.exclude(druid_task_tmp_dir);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  channels.close();\n} catch (UncheckedIOException e) {\n  IOException cause = e.getCause();\n  LOG.warn(e, \"Failed to close frame channel; checking storage health\");\n  // inspect cause: disk full / stale handle / permissions, then retry the task\n}","preventionTips":["Keep spill directories under Druid's managed task dir, excluded from external cleanup.","Monitor node disk health, free space, and open-file descriptor limits.","Prefer local reliable disks over flaky network mounts for frame spill.","Retry transient storage failures at the task level."],"tags":["io","resource-cleanup","filesystem"],"backgroundTag":"file-close-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}