apache/druid · error · IOException

Failed to fetch google cloud storage object from bucket [%s]

Error message

Failed to fetch google cloud storage object from bucket [%s] and path [%s].

What it means

GoogleStorage.size() fetches only the SIZE field of a GCS blob via the Storage client. When the object does not exist at bucket/path (or is not visible to the credentials), the SDK returns null and Druid converts that into an IOException instead of returning a size. It signals 'object not found', not a transport failure.

Source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleStorage.java:213

    if (statuses.contains(false)) {
      log.debug(
          "Google cloud storage object(s) to be deleted not found in bucket [%s].",
          bucket
      );
    }
  }

  public boolean exists(final String bucket, final String path)
  {
    Blob blob = storage.get().get(bucket, path);
    return blob != null;
  }

  public long size(final String bucket, final String path) throws IOException
  {
    Blob blob = storage.get().get(bucket, path, Storage.BlobGetOption.fields(Storage.BlobField.SIZE));
    if (blob == null) {
      throw new IOE("Failed to fetch google cloud storage object from bucket [%s] and path [%s].", bucket, path);
    }
    return blob.getSize();
  }

  /**
   * Return the etag for an object. This is a value that changes whenever the object's data or metadata changes and is
   * typically but not always the MD5 hash of the object. Ref:
   * <a href="https://cloud.google.com/storage/docs/hashes-etags#etags">ETags</a>
   *
   * @param bucket
   * @param path
   *
   * @return
   *
   * @throws IOException
   */
  public String version(final String bucket, final String path) throws IOException
  {

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the object exists with the same credentials: gsutil -o Credentials/... ls gs://<bucket>/<path> (or storage.get(bucket, path) != null) before calling size().
Check the configured bucket and key-building logic (prefix, task id substitution ':' -> '_') for mismatches.
Grant the service account roles/storage.objectViewer on the bucket.
Handle null/IOException in callers so a missing object is treated as absent rather than fatal.
If objects were legitimately deleted, refresh the metadata/segment state instead of retrying the lookup.

Example fix

// before
long size = druidStorage.size(bucket, path);
// after
Blob blob = storage.get(bucket, path, Storage.BlobGetOption.fields(Storage.BlobField.SIZE));
if (blob == null) {
  log.warn("GCS object gs://%s/%s not found, skipping", bucket, path);
  return -1;
}
long size = blob.getSize();

Defensive patterns

Strategy: validation

Validate before calling

boolean exists = storage.get(bucket, path, Storage.BlobGetOption.fields(Storage.BlobField.SIZE)) != null;
if (!exists) { /* skip or alert before calling size() */ }

Type guard

static boolean gcsObjectExists(Storage storage, String bucket, String path) {
  return storage.get(bucket, path, Storage.BlobGetOption.fields(Storage.BlobField.SIZE)) != null;
}

Try / catch

try { long size = googleStorage.size(bucket, path); }
catch (IOException e) { log.warn("GCS object gs://%s/%s missing", bucket, path); }

Prevention

Existence-check objects before size/version reads.
Keep bucket/prefix config consistent with the writer's key scheme.
Grant service accounts read access to all target buckets.
Treat missing objects as a normal state in polling code.

When it happens

Trigger: Calling GoogleStorage.size(bucket, path) for a key that was never written, was deleted, has a different case/encoding than requested, or is in a bucket the service account cannot read.

Common situations: Task log/report lookups before the task pushed its files; stale metadata pointing at segments pruned by retention policies; typos in the configured GCS bucket or prefix; service account lacking storage.objects.get on the bucket.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/b2d727b372ae1051. Report an issue: GitHub.