{"record":{"id":"b532d09d83f9a9ca","repo":"apache/hadoop","slug":"the-volume-has-already-closed","errorCode":null,"errorMessage":"The volume has already closed.","messagePattern":"The volume has already closed\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":338,"sourceCode":"  private void checkReference() {\n    Preconditions.checkState(reference.getReferenceCount() > 0);\n  }\n\n  @VisibleForTesting\n  public int getReferenceCount() {\n    return this.reference.getReferenceCount();\n  }\n\n  /**\n   * Close this volume.\n   * @throws IOException if the volume is closed.\n   */\n  void setClosed() throws IOException {\n    try {\n      this.reference.setClosed();\n      dataset.stopAllDataxceiverThreads(this);\n    } catch (ClosedChannelException e) {\n      throw new IOException(\"The volume has already closed.\", e);\n    }\n  }\n\n  /**\n   * Check whether this volume has successfully been closed.\n   */\n  boolean checkClosed() {\n    if (this.reference.getReferenceCount() > 0) {\n      FsDatasetImpl.LOG.debug(\"The reference count for {} is {}, wait to be 0.\",\n          this, reference.getReferenceCount());\n      return false;\n    }\n    return true;\n  }\n\n  @VisibleForTesting\n  public File getCurrentDir() {\n    return currentDir;","sourceCodeStart":320,"sourceCodeEnd":356,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L320-L356","documentation":"FsVolumeImpl.setClosed() flips the volume's reference count to closed and stops its dataxceiver threads. The underlying reference.setClosed() throws ClosedChannelException when the volume was already closed; this IOException wrapper surfaces a double close - typically two shutdown paths (volume eviction and dataset shutdown) racing during DN stop or disk failure handling.","triggerScenarios":"A volume fails and is evicted while DN shutdown also closes volumes; refreshNamenodes removing a block pool closing the same volume twice; test teardown double-closing.","commonSituations":"DN shutdown logs after volume failures; disk hot-remove; almost always benign noise since the second close has nothing left to do.","solutions":["If seen once during shutdown, ignore - the volume is already closed, which is the desired end state.","If it floods logs, check for duplicate shutdown hooks or monitoring tooling calling volume close twice.","Ensure volumes are evicted through one path (FsVolumeList) rather than ad-hoc closes.","Upgrade: shutdown races have been progressively quieted across releases."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  volume.setClosed();\n} catch (IOException e) {\n  if (e.getCause() instanceof ClosedChannelException) {\n    // already closed: benign double close during shutdown\n  } else {\n    throw e;\n  }\n}","preventionTips":["Close volumes through one path (FsVolumeList) rather than ad-hoc setClosed calls.","Do not register duplicate shutdown hooks that touch the same volumes.","Treat a single occurrence during shutdown as noise; only chase repeats."],"tags":["hdfs","datanode","volume","shutdown","race-condition"],"backgroundTag":"double-close-race","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}