What causes "Failed to TRUNCATE_FILE for on because is being truncated." in apache/hadoop?

Wait for the in-flight truncate/block recovery to finish (poll getFileStatus().getLen() until stable, or isFileClosed(path) true), then retry

apache/hadoop · error · AlreadyBeingCreatedException

Failed to TRUNCATE_FILE <src> for <clientName> on <clientMac

Error message

Failed to TRUNCATE_FILE <src> for <clientName> on <clientMachine> because <src> is being truncated.

What it means

Thrown by the NameNode when a client calls truncate() on a file whose last block is already UNDER_RECOVERY from another in-flight truncate with a different target length. The NameNode detects the conflicting truncation and rejects it with AlreadyBeingCreatedException, protecting the file tail from two concurrent truncates producing inconsistent state. An idempotent retry with the same target length returns success instead of throwing.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java:117

      if (lpPolicy != null && lpPolicy.getId() == file.getStoragePolicyID()) {
        throw new UnsupportedOperationException(
            "Cannot truncate lazy persist file " + src);
      }

      // Check if the file is already being truncated with the same length
      final BlockInfo last = file.getLastBlock();
      if (last != null && last.getBlockUCState()
          == BlockUCState.UNDER_RECOVERY) {
        final BlockInfo truncatedBlock = last.getUnderConstructionFeature()
            .getTruncateBlock();
        if (truncatedBlock != null) {
          final long truncateLength = file.computeFileSize(false, false)
              + truncatedBlock.getNumBytes();
          if (newLength == truncateLength) {
            return new TruncateResult(false, fsd.getAuditFileInfo(iip));
          } else {
            throw new AlreadyBeingCreatedException(
                RecoverLeaseOp.TRUNCATE_FILE.getExceptionMessage(src,
                    clientName, clientMachine, src + " is being truncated."));
          }
        }
      }

      // Opening an existing file for truncate. May need lease recovery.
      fsn.recoverLeaseInternal(RecoverLeaseOp.TRUNCATE_FILE, iip, src,
          clientName, clientMachine, false);
      // Truncate length check.
      long oldLength = file.computeFileSize();
      if (oldLength == newLength) {
        return new TruncateResult(true, fsd.getAuditFileInfo(iip));
      }
      if (oldLength < newLength) {
        throw new HadoopIllegalArgumentException(
            "Cannot truncate to a larger file size. Current size: " + oldLength
                + ", truncate size: " + newLength + ".");

View on GitHub (pinned to 2add963021)

Solutions

Wait for the in-flight truncate/block recovery to finish (poll getFileStatus().getLen() until stable, or isFileClosed(path) true), then retry
Force a stuck lease with 'hdfs debug recoverLease -path <src>' or wait past the lease soft limit (dfs.namenode.lease-soft-limit-sec, default 60 s), then retry
Serialize truncates per file in application code (single owner or lock) so only one target length is ever in flight

Example fix

// before: two writers truncate concurrently with different lengths
new Thread(() -> dfs.truncate(path, 1_000L)).start();
new Thread(() -> dfs.truncate(path, 2_000L)).start(); // AlreadyBeingCreatedException

// after: single owner; wait until the file is closed, then truncate
DistributedFileSystem dfs = (DistributedFileSystem) fs;
while (!dfs.isFileClosed(path)) {
  Thread.sleep(1_000L);
}
boolean truncated = dfs.truncate(path, newLength);
while (!truncated) { // false means block recovery in progress: re-invoke
  Thread.sleep(1_000L);
  truncated = dfs.truncate(path, newLength);
}

Defensive patterns

Strategy: retry

Validate before calling

DistributedFileSystem dfs = (DistributedFileSystem) fs;
// only truncate when no writer or recovery holds the file
if (dfs.isFileClosed(path)) {
  boolean truncated = dfs.truncate(path, newLength);
  // false => block recovery in progress, poll again
}

Try / catch

try {
  dfs.truncate(path, newLength);
} catch (AlreadyBeingCreatedException e) {
  // another truncate/recovery owns the tail: back off, re-check, retry
  Thread.sleep(2_000L);
  if (dfs.isFileClosed(path)) {
    dfs.truncate(path, newLength);
  }
}

Prevention

Route all truncates of a file through one owning process
Retry with the same newLength so the idempotent path can succeed
Check isFileClosed() before touching the tail of recently written files

When it happens

Trigger: ClientProtocol.truncate(src, newLength) while another truncate or block/lease recovery holds the last block UNDER_RECOVERY with a truncateBlock whose implied target length differs from newLength. Same newLength takes the no-op success path; a different newLength throws.

Common situations: An application restarts and retries truncate with a recalculated length while the first truncate is still settling block recovery; two jobs compacting the same file concurrently; a truncate issued right after a writer crashed while recovery is running.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d34cb55f4da961af. Report an issue: GitHub.