apache/druid · error · UncheckedIOException

Could not close channel for level [%d] and rank [%d]

Error message

Could not close channel for level [%d] and rank [%d]

What it means

When a merge step finishes, SuperSorter closes the PartitionedReadableFrameChannels it consumed. If closing one throws an IOException, it is wrapped in an UncheckedIOException identifying the merge level and rank. This indicates a resource cleanup failure — often a symptom of an underlying storage or filesystem problem rather than a bug in merge logic itself.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/processor/SuperSorter.java:783

        synchronized (runWorkersLock) {
          outputsReadyByLevel.computeIfAbsent(level, ignored2 -> new LongRBTreeSet())
                             .add(rank);
          superSorterProgressTracker.addMergedBatchesForLevel(level, 1);

          if (isLimited() && totalMergingLevels != UNKNOWN_LEVEL && level == totalMergingLevels - 1) {
            rowLimit -= outputRows;

            if (rowLimit < 0) {
              throw DruidException.defensive("rowLimit[%d] below zero after outputRows[%d]", rowLimit, outputRows);
            }
          }

          for (PartitionedReadableFrameChannel partitionedReadableFrameChannel : partitionedReadableChannelsToClose) {
            try {
              partitionedReadableFrameChannel.close();
            }
            catch (IOException e) {
              throw new UncheckedIOException(
                  StringUtils.format("Could not close channel for level [%d] and rank [%d]", level, rank),
                  e
              );
            }
          }
        }
      });
    }
    catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

  private <T> void runWorker(final FrameProcessor<T> worker, final Consumer<T> outConsumer)
  {
    Futures.addCallback(
        exec.runFully(processorDecorator.decorate(worker), cancellationId),
        new FutureCallback<>()

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect the underlying IOException cause and the affected node's disk/filesystem health (dmesg, disk space).
2. Check that the task's spill/working directories are not being modified or deleted externally while the query runs.
3. Re-run the query/task; transient storage I/O errors often clear on retry.
4. If persistent, verify filesystem permissions and mount health for druid's temporary/spill directories.

Example fix

// before (fragile cleanup location)
// spill directory shared and deleted by external cleanup cron
// after
// exclude active task tmp dirs from external cleanup, or keep spill under druid's managed task dir
cleanup_job.exclude(druid_task_tmp_dir);

try {
  channels.close();
} catch (UncheckedIOException e) {
  IOException cause = e.getCause();
  LOG.warn(e, "Failed to close frame channel; checking storage health");
  // inspect cause: disk full / stale handle / permissions, then retry the task
}

When it happens

Trigger: An IOException thrown while closing a partitioned readable frame channel in runMerger's cleanup path — typically when the channel is backed by spilled frames on disk and close() triggers file/stream cleanup that fails (disk full during deletion bookkeeping, NFS/stale file handle, file already removed, permission problems).

Common situations: Task working directory cleaned up concurrently by another process while the merger closes channels; network filesystems (HDFS/NFS/S3-backed spill) failing on close; disk-full or I/O errors on the node running the task; leaked descriptors hitting OS limits causing close failures.

Related errors

• Cannot list directory [%s]
• Unable to close channel for name :
• Path [%s] is not writable, check permissions
• Failed to create temporary directory in path [%s]
• Cannot create directory [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/bf9b53d13324c96f. Report an issue: GitHub.