apache/hadoop · error · IOException

Failed to checkpoint trash: {checkpoint}

Error message

Failed to checkpoint trash: {checkpoint}

What it means

Thrown by TrashPolicyDefault.createCheckpointImpl after more than 1000 rename attempts from .Trash/Current to the checkpoint path all failed with FileAlreadyExistsException. The checkpoint name is timestamp-based, and each collision appends -1, -2, ... -N; exhausting 1000 suffixes means the trash directory already contains a thousand checkpoints within the same timestamp window (typically because checkpoints were created but never expunged).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java:354

    if (!fs.exists(new Path(trashRoot, CURRENT))) {
      return;
    }
    Path checkpointBase;
    synchronized (CHECKPOINT) {
      checkpointBase = new Path(trashRoot, CHECKPOINT.format(date));
    }
    Path checkpoint = checkpointBase;
    Path current = new Path(trashRoot, CURRENT);

    int attempt = 0;
    while (true) {
      try {
        fs.rename(current, checkpoint, Rename.NONE);
        LOG.info("Created trash checkpoint: " + checkpoint.toUri().getPath());
        break;
      } catch (FileAlreadyExistsException e) {
        if (++attempt > 1000) {
          throw new IOException("Failed to checkpoint trash: " + checkpoint);
        }
        checkpoint = checkpointBase.suffix("-" + attempt);
      }
    }
  }

  private void deleteCheckpoint(Path trashRoot, boolean deleteImmediately)
      throws IOException {
    LOG.info("TrashPolicyDefault#deleteCheckpoint for trashRoot: " + trashRoot);

    FileStatus[] dirs = null;
    try {
      dirs = fs.listStatus(trashRoot); // scan trash sub-directories
    } catch (FileNotFoundException fnfe) {
      return;
    }

    long now = Time.now();

View on GitHub (pinned to 2add963021)

Solutions

1. Manually clear old checkpoints: hadoop fs -rm -r /user/<u>/.Trash/* (or just the dated checkpoint dirs), then retry expunge
2. Schedule regular 'hadoop fs -expunge' / trash emptier so checkpoints never accumulate to this scale
3. Verify host clocks (ntd/chrony) so timestamped checkpoint names do not collide

try {
  trash.expungeImmediately(); // or createCheckpoint()
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to checkpoint trash")) {
    // 1000+ suffix collisions: purge old checkpoints, then retry
    fs.delete(new Path(trashRoot, "Current"), true);
  }
}

When it happens

Trigger: 'hadoop fs -expunge' or FileSystem trash checkpointing on a .Trash holding 1000+ checkpoints sharing the base timestamp - e.g. expunge never ran, a job creates checkpoints in a tight loop, or system clock jumps make timestamps collide.

Common situations: Long-lived clusters where trash empties are never scheduled; automated jobs invoking expunge repeatedly; clock-skewed hosts reusing the same timestamp.

Related errors

• Failed to move {path} to trash {trashPath}
• Source path and dest path refer the same file or directory
• It is not allowed to rename a parent directory: %s to its su
• File: %s already exists
• Cannot rename %s to %s, %s is a file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a113c5b47c3f33fd. Report an issue: GitHub.