apache/hadoop · error · IOException

Cannot finalize block: {b} from Interrupted Thread

Error message

Cannot finalize block: {b} from Interrupted Thread

What it means

IOException thrown by FsDatasetImpl.finalizeBlock when the calling thread's interrupt flag is set (Thread.interrupted() returns true and clears it). The DataNode deliberately refuses data-modifying operations from interrupted threads so a thread being shut down cannot mutate on-disk state halfway. Encountered almost exclusively on DataNode shutdown/re-registration paths.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2010

  // is created but non-valid, and has been idle for >48 hours,
  // we can GC it safely.
  //

  /**
   * Complete the block write!
   */
  @Override // FsDatasetSpi
  public void finalizeBlock(ExtendedBlock b, boolean fsyncDir)
      throws IOException {
    ReplicaInfo replicaInfo = null;
    ReplicaInfo finalizedReplicaInfo = null;
    long startTimeMs = Time.monotonicNow();
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        b.getBlockPoolId(), getStorageUuidForLock(b),
        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
      if (Thread.interrupted()) {
        // Don't allow data modifications from interrupted threads
        throw new IOException("Cannot finalize block: " + b + " from Interrupted Thread");
      }
      replicaInfo = getReplicaInfo(b);
      if (replicaInfo.getState() == ReplicaState.FINALIZED) {
        // this is legal, when recovery happens on a file that has
        // been opened for append but never modified
        return;
      }
      finalizedReplicaInfo = finalizeReplica(b.getBlockPoolId(), replicaInfo);
    } finally {
      if (dataNodeMetrics != null) {
        long finalizeBlockMs = Time.monotonicNow() - startTimeMs;
        dataNodeMetrics.addFinalizeBlockOp(finalizeBlockMs);
      }
    }
    /*
     * Sync the directory after rename from tmp/rbw to Finalized if
     * configured. Though rename should be atomic operation, sync on both
     * dest and src directories are done because IOUtils.fsync() calls

View on GitHub (pinned to 2add963021)

Solutions

  1. If seen during planned restart/decommission, it is benign - the block will be finalized on the next write attempt or recovered via lease recovery.
  2. If caused by an executor shutdownNow(), drain in-flight finalize tasks before interrupting threads (graceful shutdown).
  3. Check that nothing external is sending interrupts to DataNode threads (JMX tooling, misbehaving watchdogs).
  4. For repeated occurrences outside shutdown windows, capture a thread dump to identify who interrupts the thread.
Defensive patterns

Strategy: validation

Validate before calling

// Before finalizing from an executor-managed thread:
if (Thread.currentThread().isInterrupted()) {
  throw new IllegalStateException("Refusing finalize on interrupted thread; drain task queue instead");
}

Try / catch

try {
  dataset.finalizeBlock(b, fsyncDir);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Interrupted Thread")) {
    // benign during shutdown; the block finalizes on next attempt or via lease recovery
    LOG.debug("finalize skipped: thread interrupted during shutdown", e);
    return;
  }
  throw e;
}

Prevention

When it happens

Trigger: finalizeBlock(b, fsyncDir) called from a thread that received Thread.interrupt() - typically the DataNode's shutdown hook or a BPServiceProcessor actor being cancelled during block pool restart while a client FSYNC/CLOSE request was in flight.

Common situations: DataNode shutdown or block pool re-registration racing a file close; test harnesses that interrupt DN threads to simulate failure; frameworks (e.g., ForkJoin/executor shutdownNow) that interrupt worker threads mid-operation.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e0cf68a28bb217f4. Report an issue: GitHub.