{"record":{"id":"390c2769d78fe47b","repo":"apache/druid","slug":"failed-to-fetch-google-cloud-storage-object-from-b","errorCode":null,"errorMessage":"Failed to fetch google cloud storage object from bucket [%s] and path[%s].","messagePattern":"Failed to fetch google cloud storage object from bucket \\[(.+?)\\] and path\\[(.+?)\\]\\.","errorType":"exception","errorClass":"IOException","httpStatus":500,"severity":"error","filePath":"extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleStorage.java","lineNumber":154,"sourceCode":"      final String path,\n      @Nullable final Integer chunkSize\n  )\n  {\n    WriteChannel writer = storage.get().writer(getBlobInfo(bucket, path));\n    // Limit GCS internal write buffer memory to prevent OOM errors\n    writer.setChunkSize(chunkSize == null ? DEFAULT_WRITE_CHUNK_SIZE.getBytesInInt() : chunkSize);\n\n    return Channels.newOutputStream(writer);\n  }\n\n  public GoogleStorageObjectMetadata getMetadata(\n      final String bucket,\n      final String path\n  ) throws IOException\n  {\n    Blob blob = storage.get().get(bucket, path, Storage.BlobGetOption.fields(Storage.BlobField.values()));\n    if (blob == null) {\n      throw new IOE(\"Failed to fetch google cloud storage object from bucket [%s] and path[%s].\", bucket, path);\n    }\n    return new GoogleStorageObjectMetadata(\n        blob.getBucket(),\n        blob.getName(),\n        blob.getSize(),\n        blob.getUpdateTimeOffsetDateTime()\n            .toEpochSecond() * 1000\n    );\n  }\n\n\n  /**\n   * Deletes an object in a bucket on the specified path\n   *\n   * A false response from GCS delete API is indicative of file not found. Any other error is raised as a StorageException\n   * and should be explicitly handled.\n   * Ref: <a href=\"https://github.com/googleapis/java-storage/blob/v2.29.1/google-cloud-storage/src/main/java/com/google/cloud/storage/spi/v1/HttpStorageRpc.java\">HttpStorageRpc.java</a>\n   *","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleStorage.java#L136-L172","documentation":"GoogleStorage.getMetadata fetches a GCS blob's metadata for a bucket/path and throws IOException immediately when the API returns a null Blob, meaning no object exists at that location (the GCS client returns null instead of throwing for missing objects).","triggerScenarios":"Calling getMetadata(bucket, path) for a GCS key that does not exist: segment already deleted, wrong bucket/prefix configuration, or path computed from stale segment metadata; also immediately after a failed push.","commonSituations":"Killing/reverting segments that were never pushed to this location; misconfigured druid.google.prefix; race with a concurrent kill task; metadata in the Druid metadata store pointing at cleaned-up deep storage.","solutions":["Verify the segment exists in GCS at the exact bucket/path (gsutil stat) before calling","Check that bucket and prefix configuration match the pusher's settings","Handle IOException as 'object not found' where appropriate and skip/log instead of failing","Refresh or clean stale rows in the Druid metadata store (druid_segments table)"],"exampleFix":"// before\nGoogleStorageObjectMetadata md = storage.getMetadata(bucket, path);\n// after\ntry {\n  GoogleStorageObjectMetadata md = storage.getMetadata(bucket, path);\n} catch (IOException e) {\n  LOGGER.warn(e, \"Object gs://%s/%s not found; skipping\", bucket, path);\n}","handlingStrategy":"try-catch","validationCode":"// Check existence with the raw client first\nif (storage.get(bucket, path) == null) { /* treat as not found */ }","typeGuard":null,"tryCatchPattern":"try { md = storage.getMetadata(bucket, path); } catch (IOException e) { LOGGER.warn(e, \"gs://%s/%s not found\", bucket, path); return null; }","preventionTips":["Confirm object exists (gsutil stat) before metadata fetch","Keep metadata store rows in sync with deep storage","Use the same bucket/prefix for push and fetch configs","Handle concurrent kill races gracefully"],"tags":["google-cloud-storage","resource-not-found","metadata","apache-druid"],"backgroundTag":"resource-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}