What causes "Deleted only {} of {} files from table {} using bulk deletes" in apache/iceberg?

Rerun the maintenance task (e.g. DELETE ORPHAN FILES / expire) — deletion is idempotent and remaining files will be retried on the next run.

apache/iceberg · warning

Deleted only {} of {} files from table {} using bulk deletes

Error message

Deleted only {} of {} files from table {} using bulk deletes

What it means

This is a WARN log (not a thrown exception) emitted when a bulk delete of data/delete files partially fails: a BulkDeletionFailureException reports how many objects the object store refused to delete. The processor counts the files that were actually deleted (total minus numberFailedObjects), increments the succeeded counter, and continues. It signals incomplete cleanup of orphan/obsolete files rather than a pipeline failure.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:111

  public void processWatermark(Watermark mark) {
    deleteFiles();
  }

  @Override
  public void prepareSnapshotPreBarrier(long checkpointId) {
    deleteFiles();
  }

  private void deleteFiles() {
    try {
      io.deleteFiles(filesToDelete);
      LOG.info(
          "Deleted {} files from table {} using bulk deletes", filesToDelete.size(), tableName);
      succeededCounter.inc(filesToDelete.size());
      filesToDelete.clear();
    } catch (BulkDeletionFailureException e) {
      int deletedFilesCount = filesToDelete.size() - e.numberFailedObjects();
      LOG.warn(
          "Deleted only {} of {} files from table {} using bulk deletes",
          deletedFilesCount,
          filesToDelete.size(),
          tableName,
          e);
      succeededCounter.inc(deletedFilesCount);
      failedCounter.inc(e.numberFailedObjects());
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

Rerun the maintenance task (e.g. DELETE ORPHAN FILES / expire) — deletion is idempotent and remaining files will be retried on the next run.
Check object store access: verify credentials, IAM delete permissions, bucket versioning/object-lock, and lifecycle rules that may conflict.
Inspect the BulkDeletionFailureException/underlying cause for per-object errors (throttling, NoSuchKey) and reduce batch size or add retry/backoff on the FileIO client.
Ensure only one maintenance job runs concurrently against the table to avoid double-deletion races.

Example fix

// before: bulk delete fails wholesale handling
io.deleteFilesWithBulk(locations);
// after: fall back to per-file delete for failures
try {
  io.deleteFilesWithBulk(locations);
} catch (BulkDeletionFailureException e) {
  locations.removeAll(e.failedObjects());
  for (String loc : e.failedObjects()) {
    try { io.deleteFile(loc); } catch (RuntimeException ignored) { /* retry next run */ }
  }
}

Defensive patterns

Strategy: retry

Validate before calling

// pre-check a sample location is deletable and bucket supports delete
String sample = locations.iterator().next();
try (FileIO io = table.io()) {
  Preconditions.checkArgument(io instanceof SupportsBulkOperations || locations.size() == 1,
      "FileIO %s does not support bulk deletes", io.getClass().getSimpleName());
}

Type guard

boolean supportsBulkDelete(FileIO io) { return io instanceof SupportsBulkOperations; }

Try / catch

try {
  io.deleteFilesWithBulk(locations);
} catch (BulkDeletionFailureException e) {
  // partial success; reconcile and retry failed subset idempotently
  Set<String> remaining = new HashSet<>(e.failedObjects());
  scheduleRetry(remaining);
}

Prevention

Run maintenance tasks idempotently and rerun them; partial deletes self-heal on the next run.
Use an S3/HDFS client tuned for bulk deletes (adequate concurrency, retry/throttling backoff).
Avoid overlapping maintenance jobs on the same table.
Monitor the succeededCounter vs. total to alert on repeated partial deletions.

When it happens

Trigger: deleteFiles() performs a bulk delete of a batch of files and the underlying FileIO bulk-delete call throws BulkDeletionFailureException with numberFailedObjects() > 0. Called from processElement, processWatermark, and prepareSnapshotPreBarrier while the maintenance task is expiring/cleaning files.

Common situations: S3 eventual-consistency or throttling (503 SlowDown) during large batch deletes; files already removed by a concurrent maintenance run or orphan cleanup; object-lock/retention or versioned buckets rejecting DELETE; transient credentials/permission issues on part of the batch.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0833daf3f5a7ccdf. Report an issue: GitHub.