What causes "Error while attempting to remove bucket gs://%s" in apache/beam?

Defer bucket deletion to a non-interruptible cleanup step or external job

apache/beam · error · IOException

Error while attempting to remove bucket gs://

Error message

Error while attempting to remove bucket gs://%s

What it means

GcsUtilV1.removeBucket converts an InterruptedException during the GCS bucket delete call into this IOException. A 404 is handled separately as FileNotFoundException (bucket already gone), so this error means bucket deletion was interrupted. The interrupt flag is restored before throwing.

Solutions

Defer bucket deletion to a non-interruptible cleanup step or external job
Catch IOException and detect InterruptedException cause to skip cleanup gracefully
Ignore failure if the bucket will be garbage-collected by a lifecycle policy

Example fix

// before
gcsUtil.removeBucket(bucket);
// after
try {
  gcsUtil.removeBucket(bucket);
} catch (IOException e) {
  if (e.getCause() instanceof InterruptedException) {
    Thread.currentThread().interrupt();
    LOG.warn("Bucket cleanup interrupted; relying on lifecycle policy");
    return;
  }
  throw e;
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (gcsUtil.bucketExists(bucket)) { gcsUtil.removeBucket(bucket); }

Type guard

boolean isInterruption(IOException e) { return e.getCause() instanceof InterruptedException; }

Try / catch

try {
  gcsUtil.removeBucket(bucket);
} catch (FileNotFoundException e) {
  // already deleted; ignore
} catch (IOException e) {
  if (e.getCause() instanceof InterruptedException) { Thread.currentThread().interrupt(); }
  throw e;
}

Prevention

Treat missing-bucket as success in cleanup code
Run deletion outside interruptible shutdown hooks
Use bucket lifecycle rules instead of manual cleanup

When it happens

Trigger: Calling removeBucket(bucket) while the executing thread is interrupted, e.g. cleanup running during shutdown or cancellation.

Common situations: Temp-bucket cleanup executed in a shutdown hook that gets interrupted; test teardown racing with executor termination.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/17c2d6735729e3f8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:937

              if (errorExtractor.itemNotFound(e) || errorExtractor.accessDenied(e)) {
                return false;
              }
              return RETRY_DETERMINER.shouldRetry(e);
            }
          },
          IOException.class,
          sleeper);
    } catch (GoogleJsonResponseException e) {
      if (errorExtractor.accessDenied(e)) {
        throw new AccessDeniedException(bucket.getName(), null, e.getMessage());
      }
      if (errorExtractor.itemNotFound(e)) {
        throw new FileNotFoundException(e.getMessage());
      }
      throw e;
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new IOException(
          String.format("Error while attempting to remove bucket gs://%s", bucket.getName()), e);
    }
  }

  private static void executeBatches(List<BatchInterface> batches) throws IOException {
    ExecutorService executor =
        MoreExecutors.listeningDecorator(
            new ThreadPoolExecutor(
                MAX_CONCURRENT_BATCHES,
                MAX_CONCURRENT_BATCHES,
                0L,
                TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue<>()));

    List<CompletionStage<Void>> futures = new ArrayList<>();
    for (final BatchInterface batch : batches) {
      futures.add(MoreFutures.runAsync(batch::execute, executor));
    }

View on GitHub (pinned to 12126d8942)