apache/hadoop · warning · IOException

The volume has already closed.

Error message

The volume has already closed.

What it means

FsVolumeImpl.setClosed() flips the volume's reference count to closed and stops its dataxceiver threads. The underlying reference.setClosed() throws ClosedChannelException when the volume was already closed; this IOException wrapper surfaces a double close - typically two shutdown paths (volume eviction and dataset shutdown) racing during DN stop or disk failure handling.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:338

  private void checkReference() {
    Preconditions.checkState(reference.getReferenceCount() > 0);
  }

  @VisibleForTesting
  public int getReferenceCount() {
    return this.reference.getReferenceCount();
  }

  /**
   * Close this volume.
   * @throws IOException if the volume is closed.
   */
  void setClosed() throws IOException {
    try {
      this.reference.setClosed();
      dataset.stopAllDataxceiverThreads(this);
    } catch (ClosedChannelException e) {
      throw new IOException("The volume has already closed.", e);
    }
  }

  /**
   * Check whether this volume has successfully been closed.
   */
  boolean checkClosed() {
    if (this.reference.getReferenceCount() > 0) {
      FsDatasetImpl.LOG.debug("The reference count for {} is {}, wait to be 0.",
          this, reference.getReferenceCount());
      return false;
    }
    return true;
  }

  @VisibleForTesting
  public File getCurrentDir() {
    return currentDir;

View on GitHub (pinned to 2add963021)

Solutions

1. If seen once during shutdown, ignore - the volume is already closed, which is the desired end state.
2. If it floods logs, check for duplicate shutdown hooks or monitoring tooling calling volume close twice.
3. Ensure volumes are evicted through one path (FsVolumeList) rather than ad-hoc closes.
4. Upgrade: shutdown races have been progressively quieted across releases.

try {
  volume.setClosed();
} catch (IOException e) {
  if (e.getCause() instanceof ClosedChannelException) {
    // already closed: benign double close during shutdown
  } else {
    throw e;
  }
}

When it happens

Trigger: A volume fails and is evicted while DN shutdown also closes volumes; refreshNamenodes removing a block pool closing the same volume twice; test teardown double-closing.

Common situations: DN shutdown logs after volume failures; disk hot-remove; almost always benign noise since the second close has nothing left to do.

Related errors

• Received unimplemented DNA_SHUTDOWN
• DN shut down before block pool connected
• DN shut down before block pool registered
• FsDatasetSpi has not been initialized
• Shutdown already in progress.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b532d09d83f9a9ca. Report an issue: GitHub.