{"record":{"id":"a113c5b47c3f33fd","repo":"apache/hadoop","slug":"failed-to-checkpoint-trash-checkpoint","errorCode":null,"errorMessage":"Failed to checkpoint trash: {checkpoint}","messagePattern":"Failed to checkpoint trash: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java","lineNumber":354,"sourceCode":"    if (!fs.exists(new Path(trashRoot, CURRENT))) {\n      return;\n    }\n    Path checkpointBase;\n    synchronized (CHECKPOINT) {\n      checkpointBase = new Path(trashRoot, CHECKPOINT.format(date));\n    }\n    Path checkpoint = checkpointBase;\n    Path current = new Path(trashRoot, CURRENT);\n\n    int attempt = 0;\n    while (true) {\n      try {\n        fs.rename(current, checkpoint, Rename.NONE);\n        LOG.info(\"Created trash checkpoint: \" + checkpoint.toUri().getPath());\n        break;\n      } catch (FileAlreadyExistsException e) {\n        if (++attempt > 1000) {\n          throw new IOException(\"Failed to checkpoint trash: \" + checkpoint);\n        }\n        checkpoint = checkpointBase.suffix(\"-\" + attempt);\n      }\n    }\n  }\n\n  private void deleteCheckpoint(Path trashRoot, boolean deleteImmediately)\n      throws IOException {\n    LOG.info(\"TrashPolicyDefault#deleteCheckpoint for trashRoot: \" + trashRoot);\n\n    FileStatus[] dirs = null;\n    try {\n      dirs = fs.listStatus(trashRoot); // scan trash sub-directories\n    } catch (FileNotFoundException fnfe) {\n      return;\n    }\n\n    long now = Time.now();","sourceCodeStart":336,"sourceCodeEnd":372,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java#L336-L372","documentation":"Thrown by TrashPolicyDefault.createCheckpointImpl after more than 1000 rename attempts from .Trash/Current to the checkpoint path all failed with FileAlreadyExistsException. The checkpoint name is timestamp-based, and each collision appends -1, -2, ... -N; exhausting 1000 suffixes means the trash directory already contains a thousand checkpoints within the same timestamp window (typically because checkpoints were created but never expunged).","triggerScenarios":"'hadoop fs -expunge' or FileSystem trash checkpointing on a .Trash holding 1000+ checkpoints sharing the base timestamp - e.g. expunge never ran, a job creates checkpoints in a tight loop, or system clock jumps make timestamps collide.","commonSituations":"Long-lived clusters where trash empties are never scheduled; automated jobs invoking expunge repeatedly; clock-skewed hosts reusing the same timestamp.","solutions":["Manually clear old checkpoints: hadoop fs -rm -r /user/<u>/.Trash/* (or just the dated checkpoint dirs), then retry expunge","Schedule regular 'hadoop fs -expunge' / trash emptier so checkpoints never accumulate to this scale","Verify host clocks (ntd/chrony) so timestamped checkpoint names do not collide"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  trash.expungeImmediately(); // or createCheckpoint()\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Failed to checkpoint trash\")) {\n    // 1000+ suffix collisions: purge old checkpoints, then retry\n    fs.delete(new Path(trashRoot, \"Current\"), true);\n  }\n}","preventionTips":["Schedule periodic expunge so checkpoint counts stay in the dozens, not thousands","Keep host clocks synchronized so timestamped checkpoint names do not collide"],"tags":["hadoop","trash","checkpoint","rename"],"backgroundTag":"trash-checkpoint-collision","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}