apache/hadoop · error · IOException

Cannot finalize file {} because it is not under construction

Error message

Cannot finalize file {} because it is not under construction

What it means

finalizeINodeFileUnderConstruction throws IOException when the INodeFile no longer carries a FileUnderConstructionFeature, meaning it was already finalized or was never opened for writing. The method only converts open files to complete files, so a second finalize attempt is rejected as an invalid state transition.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:3985

    if (i < 0) {
      i = 0;
    }
    for(; i < blocks.length; i++) {
      final BlockInfo b = blocks[i];
      if (b != null && b.getBlockUCState() == BlockUCState.COMMITTED) {
        // b is COMMITTED but not yet COMPLETE, add it to pending replication.
        blockManager.addExpectedReplicasToPending(b);
      }
    }
  }

  void finalizeINodeFileUnderConstruction(String src, INodeFile pendingFile,
      int latestSnapshot, boolean allowCommittedBlock) throws IOException {
    assert hasWriteLock(RwLockMode.GLOBAL);

    FileUnderConstructionFeature uc = pendingFile.getFileUnderConstructionFeature();
    if (uc == null) {
      throw new IOException("Cannot finalize file " + src
          + " because it is not under construction");
    }

    pendingFile.recordModification(latestSnapshot);

    // The file is no longer pending.
    // Create permanent INode, update blocks. No need to replace the inode here
    // since we just remove the uc feature from pendingFile
    pendingFile.toCompleteFile(now(),
        allowCommittedBlock? numCommittedAllowed: 0,
        blockManager.getMinReplication());

    leaseManager.removeLease(uc.getClientName(), pendingFile);

    // close file and persist block allocations for this file
    closeFile(src, pendingFile);

    blockManager.checkRedundancy(pendingFile);

View on GitHub (pinned to 2add963021)

Solutions

1. Before propagating the failure, re-check the file state: if it is closed with the expected length, treat the completion as successful (idempotent close)
2. Make client completeFile logic idempotent: on IOException, call getFileStatus and accept closed+full-length as success
3. Upgrade Hadoop if running an old 2.x line with known complete-vs-recovery races
4. Avoid immediate tight retries on completeFile; add a state check between attempts

Example fix

// before
if (!dfsClient.complete(src, clientName)) { throw new IOException("complete failed"); }

// after
if (!dfsClient.complete(src, clientName)) {
  HdfsFileStatus st = dfsClient.getFileInfo(src);
  if (st == null || !st.isClosed() || st.getLen() != expectedLen) {
    throw new IOException("complete failed for " + src);
  } // else: already finalized by a retry, accept it
}

HdfsFileStatus st = dfsClient.getFileInfo(src);
if (st != null && st.isClosed() && st.getLen() == expectedLen) {
  return; // already finalized; skip complete call
}

try {
  dfsClient.complete(src, clientName);
} catch (IOException e) {
  HdfsFileStatus st = dfsClient.getFileInfo(src);
  if (st == null || !st.isClosed()) { throw e; } // genuine failure
  // else: double-finalize race, accept as success
}

When it happens

Trigger: completeFile retried after a timeout when the first call already finalized the file; commitBlockSynchronization racing a lease recovery that closed the file first.

Common situations: Aggressive hand-rolled retry loops around completeFile; NameNode failover replaying an operation that had already succeeded; internal races fixed in later Hadoop versions.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Unexpected BlockUCState: {} is {} but not UNDER_CONSTRUCTION
• Transition from state {} to {} is not allowed.
• Unknown nameservice: {}
• Configuration has multiple addresses that match local node's
• Cannot delete/rename non-empty protected directory {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cdc51f51d5757bb0. Report an issue: GitHub.