What causes "Exception occurred while closing channel '%s'" in apache/hadoop?

Inspect the cause: if all reads already completed successfully, log and continue — the exception is usually benign at that point.

apache/hadoop · warning · IOException

Exception occurred while closing channel '%s'

Error message

Exception occurred while closing channel '%s'

What it means

GoogleCloudStorageClientReadChannel.close() wraps any exception thrown by contentReadChannel.closeContentChannel() into IOException "Exception occurred while closing channel '<resourceId>'". The finally block still nulls the channel and clears the open flag, so the channel ends up closed even when the underlying HTTP stream close failed; the original failure is preserved as the cause. Whether this matters depends on whether the read side already succeeded.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientReadChannel.java:171

  @Override
  public SeekableByteChannel truncate(long size) throws IOException {
    throw new UnsupportedOperationException("Cannot mutate read-only channel");
  }

  @Override
  public boolean isOpen() {
    return open;
  }

  @Override
  public void close() throws IOException {
    if (open) {
      try {
        LOG.trace("Closing channel for '{}'", resourceId);
        contentReadChannel.closeContentChannel();
      } catch (Exception e) {
        throw new IOException(
            String.format("Exception occurred while closing channel '%s'", resourceId), e);
      } finally {
        contentReadChannel = null;
        open = false;
      }
    }
  }

  /**
   * This class own the responsibility of opening up contentChannel. It also implements the Fadvise,
   * which helps in deciding the boundaries of content channel being opened and also caching the
   * footer of an object.
   */
  private class ContentReadChannel {

    // Size of buffer to allocate for skipping bytes in-place when performing in-place seeks.
    private static final int SKIP_BUFFER_SIZE = 8192;
    private final BlobId blobId;

View on GitHub (pinned to 2add963021)

Solutions

Inspect the cause: if all reads already completed successfully, log and continue — the exception is usually benign at that point.
Structure code so a close-time failure never masks the primary result: track whether the body was fully read, and only propagate close errors when it wasn't.
If close errors correlate with read errors or one specific host, fix the underlying network/proxy/idle-timeout problem.
Upgrade the connector and google-cloud-storage client, which have received close-robustness fixes.

Example fix

// before
ch.close(); // close failure masks the successful read result

// after
boolean readComplete = /* all bytes consumed */;
try {
  ch.close();
} catch (IOException closeErr) {
  if (!readComplete) throw closeErr;
  LOG.warn("Ignoring close failure after complete read: {}", closeErr.toString());
}

Defensive patterns

Strategy: try-catch

Try / catch

boolean readComplete = bytesConsumed == expectedBytes;
try {
  ch.close();
} catch (IOException closeErr) {
  if (!readComplete) throw closeErr;      // data may be missing: propagate
  LOG.warn("close failed after complete read: {}", closeErr.toString()); // benign
}

Prevention

Track whether the body was fully read so close failures can be classified benign vs data-loss.
Never let a close exception replace an earlier, more meaningful read exception — close guarded inside catch/finally.
Keep client-side keep-alive/proxy idle timeouts above expected read gaps to avoid half-dead connections at close.

When it happens

Trigger: Closing a read channel whose underlying storage ReadChannel/HTTP connection errors during close: connection reset by peer or proxy, connection already killed by an idle timeout (e.g. LB ~10 min), or closing after a prior read error left the stream broken.

Common situations: Flaky networks where the final FIN/RST fails; Spark/Hadoop task cleanup closing thousands of channels during network degradation; connections reaped by middleboxes; double-close after a failed read.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1d08abb7836e9aff. Report an issue: GitHub.