apache/hadoop · error · IOException

detachBlock:Block not found. ${this}

Error message

detachBlock:Block not found. ${this}

What it means

breakHardLinksIfNeeded() (detach) requires a resolvable local block file and volume; if getBlockFile() or getVolume() returns null it throws IOException('detachBlock:Block not found. ' + this). The replica has no local file binding to break links on — typically a lost volume reference or a replica type that carries no local file.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java:238

   * When doing a DataNode upgrade, we create a bunch of hardlinks to each block
   * file.  This cleverly ensures that both the old and the new storage
   * directories can contain the same block file, without using additional space
   * for the data.
   *
   * However, when we want to append to the replica file, we need to "break" the
   * hardlink to ensure that the old snapshot continues to contain the old data
   * length.  If we failed to do that, we could roll back to the previous/
   * directory during a downgrade, and find that the block contents were longer
   * than they were at the time of upgrade.
   *
   * @return true only if data was copied.
   * @throws IOException
   */
  public boolean breakHardLinksIfNeeded() throws IOException {
    final File file = getBlockFile();
    final FileIoProvider fileIoProvider = getFileIoProvider();
    if (file == null || getVolume() == null) {
      throw new IOException("detachBlock:Block not found. " + this);
    }
    File meta = getMetaFile();

    int linkCount = fileIoProvider.getHardLinkCount(getVolume(), file);
    if (linkCount > 1) {
      DataNode.LOG.info("Breaking hardlink for " + linkCount + "x-linked " +
          "block " + this);
      breakHardlinks(file, this);
    }
    if (fileIoProvider.getHardLinkCount(getVolume(), meta) > 1) {
      breakHardlinks(meta, this);
    }
    return true;
  }

  @Override
  public URI getBlockURI() {
    return getBlockFile().toURI();

View on GitHub (pinned to 2add963021)

Solutions

  1. Check whether the replica's volume is still present in DataNode volume state (JMX FSDatasetStateInfo); fix the volume (remount/replace) before retrying the upgrade.
  2. Restart the DataNode so volume and replica state is rebuilt consistently, then retry the operation.
  3. Ensure the detach flow skips replicas without local files (e.g., PROVIDED).
  4. If the block is effectively gone, let HDFS re-replicate it (verify with `hdfs fsck /`) before continuing the upgrade.

Example fix

// before: detach every replica unconditionally
for (ReplicaInfo r : replicas) {
  r.breakHardLinksIfNeeded();
}

// after: skip replicas with no local file binding
for (ReplicaInfo r : replicas) {
  if (r.getBlockFile() != null && r.getVolume() != null) {
    r.breakHardLinksIfNeeded();
  }
}
Defensive patterns

Strategy: validation

Validate before calling

if (replica.getBlockFile() != null && replica.getVolume() != null) {
  replica.breakHardLinksIfNeeded();
} else {
  // no local binding: skip (provided replica) or resolve the missing volume first
}

Try / catch

try {
  replica.breakHardLinksIfNeeded();
} catch (IOException e) {
  if (e.getMessage().contains("detachBlock:Block not found")) {
    // volume lost or file unresolved: fix volume state / restart DN before retrying
  }
}

Prevention

When it happens

Trigger: Calling breakHardLinksIfNeeded() on a replica whose volume was removed (failed disk, hot-unplug) or whose block file never resolved — e.g., provided replicas or replicas reconstructed after a restart without volume binding.

Common situations: Hard-link detach during an upgrade on a node where a volume just failed or was unmounted; DataNode restart mid-upgrade leaving replicas with null volume references; provided-storage replicas entering local-only detach flows.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/869beb6332dd4b8f. Report an issue: GitHub.