apache/beam · error · IOException

dynamic: wrapped StorageException via translateStorageExcept

Error message

dynamic: wrapped StorageException via translateStorageException (AccessDeniedException on 403, FileNotFoundException on 404, FileAlreadyExistsException on 409, else IOException)

What it means

GcsUtilV2.getBlob fetches a GCS blob via the Storage client; a null blob is reported as FileNotFoundException, and any StorageException is translated by translateStorageException into AccessDeniedException (403), FileNotFoundException (404), FileAlreadyExistsException (409), or a generic IOException. This normalizes GCS API errors into standard java.nio.file exceptions.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV2.java:134

        return new IOException(e);
    }
  }

  private IOException translateStorageException(
      String bucketName, @Nullable String blobName, StorageException e) {
    return translateStorageException(GcsPath.fromComponents(bucketName, blobName), e);
  }

  public Blob getBlob(GcsPath gcsPath, BlobGetOption... options) throws IOException {
    try {
      Blob blob = storage.get(gcsPath.getBucket(), gcsPath.getObject(), options);
      if (blob == null) {
        throw new FileNotFoundException(
            String.format("The specified file does not exist: %s", gcsPath.toString()));
      }
      return blob;
    } catch (StorageException e) {
      throw translateStorageException(gcsPath, e);
    }
  }

  public long fileSize(GcsPath gcsPath) throws IOException {
    return getBlob(gcsPath, BlobGetOption.fields(BlobField.SIZE)).getSize();
  }

  /** A class that holds either a {@link Blob} or an {@link IOException}. */
  @AutoValue
  public abstract static class BlobResult {

    /** Returns the {@link Blob}. */
    public abstract @Nullable Blob blob();

    /** Returns the {@link IOException}. */
    public abstract @Nullable IOException ioException();

    @VisibleForTesting

View on GitHub (pinned to 12126d8942)

Solutions

Verify the gs:// path/object exists with gsutil ls.
Grant the caller's service account storage.objects.get (roles/storage.objectViewer).
Catch the specific nio exceptions (FileNotFoundException vs AccessDeniedException) and act accordingly; retry IOException for transient 5xx.
Check bucket location/firewall/VPC-SC boundaries if requests are blocked.

Example fix

// before
long size = gcsUtil.fileSize(GcsPath.fromUri(path)); // throws raw
// after
try {
  long size = gcsUtil.fileSize(GcsPath.fromUri(path));
} catch (FileNotFoundException e) {
  // skip missing input
} catch (AccessDeniedException e) {
  // fix IAM / credentials
}

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check existence and permission
try {
  gcsUtil.fileSize(gcsPath);
} catch (FileNotFoundException e) { /* handle missing */ }
catch (AccessDeniedException e) { /* fix IAM before pipeline run */ }

Try / catch

try {
  Blob b = gcsUtil.blob(gcsPath);
} catch (FileNotFoundException e) {
  // object missing: skip or fail fast
} catch (AccessDeniedException e) {
  // IAM problem: alert operator
} catch (IOException e) {
  // transient: retry with backoff
}

Prevention

Validate gs:// paths and object existence before pipeline submission.
Grant least-privilege objectViewer IAM to the worker service account.
Add retries around transient IOExceptions.

When it happens

Trigger: Calling fileSize(gcsPath), blob(gcsPath,...), or expand over a GcsPath where the object does not exist, the credential lacks storage.objects.get, or the Storage client raises an error (quota, server error).

Common situations: Typo'd gs:// path or missing object; service account missing Storage Object Viewer role; bucket in another project/network blocked; transient 5xx surfaced as IOException.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/dcb646af7e643a16. Report an issue: GitHub.