apache/druid · error · IOException

Failed to fetch google cloud storage object from bucket [%s]

Error message

Failed to fetch google cloud storage object from bucket [%s] and path[%s].

What it means

GoogleStorage.getMetadata fetches a GCS blob's metadata for a bucket/path and throws IOException immediately when the API returns a null Blob, meaning no object exists at that location (the GCS client returns null instead of throwing for missing objects).

Source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleStorage.java:154

      final String path,
      @Nullable final Integer chunkSize
  )
  {
    WriteChannel writer = storage.get().writer(getBlobInfo(bucket, path));
    // Limit GCS internal write buffer memory to prevent OOM errors
    writer.setChunkSize(chunkSize == null ? DEFAULT_WRITE_CHUNK_SIZE.getBytesInInt() : chunkSize);

    return Channels.newOutputStream(writer);
  }

  public GoogleStorageObjectMetadata getMetadata(
      final String bucket,
      final String path
  ) throws IOException
  {
    Blob blob = storage.get().get(bucket, path, Storage.BlobGetOption.fields(Storage.BlobField.values()));
    if (blob == null) {
      throw new IOE("Failed to fetch google cloud storage object from bucket [%s] and path[%s].", bucket, path);
    }
    return new GoogleStorageObjectMetadata(
        blob.getBucket(),
        blob.getName(),
        blob.getSize(),
        blob.getUpdateTimeOffsetDateTime()
            .toEpochSecond() * 1000
    );
  }


  /**
   * Deletes an object in a bucket on the specified path
   *
   * A false response from GCS delete API is indicative of file not found. Any other error is raised as a StorageException
   * and should be explicitly handled.
   * Ref: <a href="https://github.com/googleapis/java-storage/blob/v2.29.1/google-cloud-storage/src/main/java/com/google/cloud/storage/spi/v1/HttpStorageRpc.java">HttpStorageRpc.java</a>
   *

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the segment exists in GCS at the exact bucket/path (gsutil stat) before calling
Check that bucket and prefix configuration match the pusher's settings
Handle IOException as 'object not found' where appropriate and skip/log instead of failing
Refresh or clean stale rows in the Druid metadata store (druid_segments table)

Example fix

// before
GoogleStorageObjectMetadata md = storage.getMetadata(bucket, path);
// after
try {
  GoogleStorageObjectMetadata md = storage.getMetadata(bucket, path);
} catch (IOException e) {
  LOGGER.warn(e, "Object gs://%s/%s not found; skipping", bucket, path);
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Check existence with the raw client first
if (storage.get(bucket, path) == null) { /* treat as not found */ }

Try / catch

try { md = storage.getMetadata(bucket, path); } catch (IOException e) { LOGGER.warn(e, "gs://%s/%s not found", bucket, path); return null; }

Prevention

Confirm object exists (gsutil stat) before metadata fetch
Keep metadata store rows in sync with deep storage
Use the same bucket/prefix for push and fetch configs
Handle concurrent kill races gracefully

When it happens

Trigger: Calling getMetadata(bucket, path) for a GCS key that does not exist: segment already deleted, wrong bucket/prefix configuration, or path computed from stale segment metadata; also immediately after a failed push.

Common situations: Killing/reverting segments that were never pushed to this location; misconfigured druid.google.prefix; race with a concurrent kill task; metadata in the Druid metadata store pointing at cleaned-up deep storage.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/390c2769d78fe47b. Report an issue: GitHub.