{"record":{"id":"e0cf68a28bb217f4","repo":"apache/hadoop","slug":"cannot-finalize-block-b-from-interrupted-thread","errorCode":null,"errorMessage":"Cannot finalize block: {b} from Interrupted Thread","messagePattern":"Cannot finalize block: (.+?) from Interrupted Thread","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":2010,"sourceCode":"  // is created but non-valid, and has been idle for >48 hours,\n  // we can GC it safely.\n  //\n\n  /**\n   * Complete the block write!\n   */\n  @Override // FsDatasetSpi\n  public void finalizeBlock(ExtendedBlock b, boolean fsyncDir)\n      throws IOException {\n    ReplicaInfo replicaInfo = null;\n    ReplicaInfo finalizedReplicaInfo = null;\n    long startTimeMs = Time.monotonicNow();\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n        b.getBlockPoolId(), getStorageUuidForLock(b),\n        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n      if (Thread.interrupted()) {\n        // Don't allow data modifications from interrupted threads\n        throw new IOException(\"Cannot finalize block: \" + b + \" from Interrupted Thread\");\n      }\n      replicaInfo = getReplicaInfo(b);\n      if (replicaInfo.getState() == ReplicaState.FINALIZED) {\n        // this is legal, when recovery happens on a file that has\n        // been opened for append but never modified\n        return;\n      }\n      finalizedReplicaInfo = finalizeReplica(b.getBlockPoolId(), replicaInfo);\n    } finally {\n      if (dataNodeMetrics != null) {\n        long finalizeBlockMs = Time.monotonicNow() - startTimeMs;\n        dataNodeMetrics.addFinalizeBlockOp(finalizeBlockMs);\n      }\n    }\n    /*\n     * Sync the directory after rename from tmp/rbw to Finalized if\n     * configured. Though rename should be atomic operation, sync on both\n     * dest and src directories are done because IOUtils.fsync() calls","sourceCodeStart":1992,"sourceCodeEnd":2028,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1992-L2028","documentation":"IOException thrown by FsDatasetImpl.finalizeBlock when the calling thread's interrupt flag is set (Thread.interrupted() returns true and clears it). The DataNode deliberately refuses data-modifying operations from interrupted threads so a thread being shut down cannot mutate on-disk state halfway. Encountered almost exclusively on DataNode shutdown/re-registration paths.","triggerScenarios":"finalizeBlock(b, fsyncDir) called from a thread that received Thread.interrupt() - typically the DataNode's shutdown hook or a BPServiceProcessor actor being cancelled during block pool restart while a client FSYNC/CLOSE request was in flight.","commonSituations":"DataNode shutdown or block pool re-registration racing a file close; test harnesses that interrupt DN threads to simulate failure; frameworks (e.g., ForkJoin/executor shutdownNow) that interrupt worker threads mid-operation.","solutions":["If seen during planned restart/decommission, it is benign - the block will be finalized on the next write attempt or recovered via lease recovery.","If caused by an executor shutdownNow(), drain in-flight finalize tasks before interrupting threads (graceful shutdown).","Check that nothing external is sending interrupts to DataNode threads (JMX tooling, misbehaving watchdogs).","For repeated occurrences outside shutdown windows, capture a thread dump to identify who interrupts the thread."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before finalizing from an executor-managed thread:\nif (Thread.currentThread().isInterrupted()) {\n  throw new IllegalStateException(\"Refusing finalize on interrupted thread; drain task queue instead\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  dataset.finalizeBlock(b, fsyncDir);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Interrupted Thread\")) {\n    // benign during shutdown; the block finalizes on next attempt or via lease recovery\n    LOG.debug(\"finalize skipped: thread interrupted during shutdown\", e);\n    return;\n  }\n  throw e;\n}","preventionTips":["Shut down executors gracefully (shutdown + awaitTermination) before interrupting threads that touch the dataset.","Do not call DataNode-internal dataset APIs from custom interrupt-happy watchdog threads.","Treat occurrences outside restart windows as a signal that something interrupts DN threads; take a thread dump."],"tags":["hdfs","datanode","finalize","thread-interrupt","shutdown"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}