apache/beam · error · IOException

Error while attempting to verify existence of bucket gs://

Error message

Error while attempting to verify existence of bucket gs://%s

What it means

GcsUtilV1.bucketExists (bucket verification) converts an InterruptedException during the GCS bucket get call into this IOException. A 404 is handled separately as FileNotFoundException (bucket not found), so this error means the existence check itself was interrupted. The thread's interrupt flag is restored before throwing.

Solutions

Avoid interrupting threads during pipeline setup/teardown, or verify the bucket earlier
Catch IOException and check getCause() instanceof InterruptedException to distinguish this case
Re-run the existence check on a non-interrupted thread if needed

Example fix

// before
if (!gcsUtil.bucketExists(path)) { ... }
// after
try {
  if (!gcsUtil.bucketExists(path)) { ... }
} catch (IOException e) {
  if (e.getCause() instanceof InterruptedException) {
    Thread.currentThread().interrupt(); // preserve status, abort setup
  }
  throw e;
}

Defensive patterns

Strategy: try-catch

Type guard

boolean isInterruption(IOException e) { return e.getCause() instanceof InterruptedException; }

Try / catch

try {
  boolean exists = gcsUtil.bucketExists(path);
} catch (IOException e) {
  if (e.getCause() instanceof InterruptedException) { Thread.currentThread().interrupt(); }
  throw e;
}

Prevention

Perform existence checks before cancellation-prone phases
Preserve the interrupt status when handling
Don't swallow InterruptedException in wrapper code

When it happens

Trigger: Calling bucketExists(gcsPath) (directly or via GCS utilities like makeGcsUtil) while the executing thread is interrupted, e.g. during pipeline shutdown or task cancellation.

Common situations: Pipelines being cancelled or drained while setup code verifies the staging bucket; executor shutdown interrupting worker threads mid-call.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a73fe29002f4551a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:860

              if (errorExtractor.itemNotFound(e) || errorExtractor.accessDenied(e)) {
                return false;
              }
              return RETRY_DETERMINER.shouldRetry(e);
            }
          },
          IOException.class,
          sleeper);
    } catch (GoogleJsonResponseException e) {
      if (errorExtractor.accessDenied(e)) {
        throw new AccessDeniedException(path.toString(), null, e.getMessage());
      }
      if (errorExtractor.itemNotFound(e)) {
        throw new FileNotFoundException(e.getMessage());
      }
      throw e;
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new IOException(
          String.format(
              "Error while attempting to verify existence of bucket gs://%s", path.getBucket()),
          e);
    }
  }

  @VisibleForTesting
  void createBucket(String projectId, Bucket bucket, BackOff backoff, Sleeper sleeper)
      throws IOException {
    Storage.Buckets.Insert insertBucket = storageClient.buckets().insert(projectId, bucket);
    insertBucket.setPredefinedAcl("projectPrivate");
    insertBucket.setPredefinedDefaultObjectAcl("projectPrivate");

    try {
      ResilientOperation.retry(
          insertBucket::execute,
          backoff,
          new RetryDeterminer<IOException>() {

View on GitHub (pinned to 12126d8942)