apache/iceberg · error · IllegalArgumentException

Cannot remove orphan files with an interval less than 24 hou

Error message

Cannot remove orphan files with an interval less than 24 hours. Executing this procedure with a short interval may corrupt the table if other operations are happening at the same time. If you are absolutely confident that no concurrent operations will be affected by removing orphan files with such a short interval, you can use the Action API to remove orphan files with an arbitrary interval.

What it means

RemoveOrphanFilesProcedure refuses to delete files newer than 24 hours. validateInterval computes now - older_than and throws IllegalArgumentException when it is under one day, because concurrent operations may still be writing files that would be deleted, corrupting the table.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/procedures/RemoveOrphanFilesProcedure.java:229

  private InternalRow[] toOutputRows(DeleteOrphanFiles.Result result) {
    Iterable<String> orphanFileLocations = result.orphanFileLocations();

    int orphanFileLocationsCount = Iterables.size(orphanFileLocations);
    InternalRow[] rows = new InternalRow[orphanFileLocationsCount];

    int index = 0;
    for (String fileLocation : orphanFileLocations) {
      rows[index] = newInternalRow(UTF8String.fromString(fileLocation));
      index++;
    }

    return rows;
  }

  private void validateInterval(long olderThanMillis) {
    long intervalMillis = System.currentTimeMillis() - olderThanMillis;
    if (intervalMillis < TimeUnit.DAYS.toMillis(1)) {
      throw new IllegalArgumentException(
          "Cannot remove orphan files with an interval less than 24 hours. Executing this "
              + "procedure with a short interval may corrupt the table if other operations are happening "
              + "at the same time. If you are absolutely confident that no concurrent operations will be "
              + "affected by removing orphan files with such a short interval, you can use the Action API "
              + "to remove orphan files with an arbitrary interval.");
    }
  }

  @Override
  public String name() {
    return NAME;
  }

  @Override
  public String description() {
    return "RemoveOrphanFilesProcedure";
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Use an interval of at least 24 hours: older_than => current_timestamp() - INTERVAL 1 DAYS
2. If you truly must use a shorter interval, call the Action API directly (SparkActions.get().deleteOrphanFiles(table).olderThan(millis)) after confirming no concurrent writes
3. Dry-run first with dry_run => true to see what would be deleted
4. Ensure clocks/timezones are consistent so the intended cutoff is actually >= 24h

Example fix

// before
CALL cat.system.remove_orphan_files(table => 'cat.db.t', older_than => current_timestamp() - INTERVAL 1 HOURS);
// after
CALL cat.system.remove_orphan_files(table => 'cat.db.t', older_than => current_timestamp() - INTERVAL 1 DAYS, dry_run => true);

if (System.currentTimeMillis() - olderThanMillis < TimeUnit.DAYS.toMillis(1)) throw new IllegalArgumentException("older_than must be at least 24 hours in the past")

try { spark.sql(call) } catch { case e: IllegalArgumentException if e.getMessage.contains("interval less than 24 hours") => /* widen interval or use Action API */ }

When it happens

Trigger: CALL ... remove_orphan_files with older_than => TIMESTAMP/INTERVAL making the cutoff less than 24 hours in the past (e.g. older_than => current_timestamp() - INTERVAL 1 HOUR).

Common situations: Trying to reclaim disk space quickly after a test; misunderstanding that older_than is an absolute timestamp (an ambiguous timezone can also shrink the effective interval); automated cleanup scripts with aggressive intervals.

Related errors

• Cannot remove orphan files with an interval less than 24 hou
• Couldn't load table '${ident}' in catalog '${tableCatalog.na
• Cannot parse predicates in where option: ${where}
• Cannot apply non-unique WAP ID. Found multiple snapshots wit
• Cannot apply unknown WAP ID '${wapId}'

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/a5e34df8c65fbe84. Report an issue: GitHub.