apache/druid · error · IOException

Failed to fetch google cloud storage object from bucket

Error message

Failed to fetch google cloud storage object from bucket [%s] and path[%s].

What it means

GoogleStorage.getMetadata fetches a GCS blob's metadata for a bucket/path and throws IOException immediately when the API returns a null Blob, meaning no object exists at that location (the GCS client returns null instead of throwing for missing objects).

Solutions

1. Verify the segment exists in GCS at the exact bucket/path (gsutil stat) before calling
2. Check that bucket and prefix configuration match the pusher's settings
3. Handle IOException as 'object not found' where appropriate and skip/log instead of failing
4. Refresh or clean stale rows in the Druid metadata store (druid_segments table)

Example fix

// before
GoogleStorageObjectMetadata md = storage.getMetadata(bucket, path);
// after
try {
  GoogleStorageObjectMetadata md = storage.getMetadata(bucket, path);
} catch (IOException e) {
  LOGGER.warn(e, "Object gs://%s/%s not found; skipping", bucket, path);
}

// Check existence with the raw client first
if (storage.get(bucket, path) == null) { /* treat as not found */ }

try { md = storage.getMetadata(bucket, path); } catch (IOException e) { LOGGER.warn(e, "gs://%s/%s not found", bucket, path); return null; }

When it happens

Trigger: Calling getMetadata(bucket, path) for a GCS key that does not exist: segment already deleted, wrong bucket/prefix configuration, or path computed from stale segment metadata; also immediately after a failed push.

Common situations: Killing/reverting segments that were never pushed to this location; misconfigured druid.google.prefix; race with a concurrent kill task; metadata in the Druid metadata store pointing at cleaned-up deep storage.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Cannot delete all segment files from Google Deep Storage…
• Couldn't kill segment
• Failed to delete google cloud storage object from bucket
• Failed to upload [ ] to [ ]
• Invalid input file path

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/390c2769d78fe47b. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleStorage.java:154

      final String path,
      @Nullable final Integer chunkSize
  )
  {
    WriteChannel writer = storage.get().writer(getBlobInfo(bucket, path));
    // Limit GCS internal write buffer memory to prevent OOM errors
    writer.setChunkSize(chunkSize == null ? DEFAULT_WRITE_CHUNK_SIZE.getBytesInInt() : chunkSize);

    return Channels.newOutputStream(writer);
  }

  public GoogleStorageObjectMetadata getMetadata(
      final String bucket,
      final String path
  ) throws IOException
  {
    Blob blob = storage.get().get(bucket, path, Storage.BlobGetOption.fields(Storage.BlobField.values()));
    if (blob == null) {
      throw new IOE("Failed to fetch google cloud storage object from bucket [%s] and path[%s].", bucket, path);
    }
    return new GoogleStorageObjectMetadata(
        blob.getBucket(),
        blob.getName(),
        blob.getSize(),
        blob.getUpdateTimeOffsetDateTime()
            .toEpochSecond() * 1000
    );
  }


  /**
   * Deletes an object in a bucket on the specified path
   *
   * A false response from GCS delete API is indicative of file not found. Any other error is raised as a StorageException
   * and should be explicitly handled.
   * Ref: <a href="https://github.com/googleapis/java-storage/blob/v2.29.1/google-cloud-storage/src/main/java/com/google/cloud/storage/spi/v1/HttpStorageRpc.java">HttpStorageRpc.java</a>
   *

View on GitHub (pinned to 9b90983fd2)