{"record":{"id":"a9a402db98174688","repo":"apache/druid","slug":"couldn-t-kill-segment-s-s-a9a402","errorCode":null,"errorMessage":"Couldn't kill segment[%s]: [%s]","messagePattern":"Couldn't kill segment\\[(.+?)\\]: \\[(.+?)\\]","errorType":"exception","errorClass":"SegmentLoadingException","httpStatus":null,"severity":"error","filePath":"extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java","lineNumber":73,"sourceCode":"\n  @Override\n  public void kill(DataSegment segment) throws SegmentLoadingException\n  {\n    LOG.info(\"Killing segment [%s]\", segment.getId());\n\n    Map<String, Object> loadSpec = segment.getLoadSpec();\n    final String bucket = MapUtils.getString(loadSpec, \"bucket\");\n    final String indexPath = MapUtils.getString(loadSpec, \"path\");\n    final String descriptorPath = DataSegmentKiller.descriptorPath(indexPath);\n\n    try {\n      deleteIfPresent(bucket, indexPath);\n      // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored\n      // anymore, but we still delete them if exists.\n      deleteIfPresent(bucket, descriptorPath);\n    }\n    catch (StorageException e) {\n      throw new SegmentLoadingException(e, \"Couldn't kill segment[%s]: [%s]\", segment.getId(), e.getMessage());\n    }\n  }\n\n  private void deleteIfPresent(String bucket, String path)\n  {\n    try {\n      GoogleUtils.retryGoogleCloudStorageOperation(() -> {\n        storage.delete(bucket, path);\n        return null;\n      });\n    }\n    catch (StorageException e) {\n      throw e;\n    }\n    catch (Exception e) {\n      throw new RE(e, \"Failed to delete google cloud storage object from bucket [%s] and path [%s].\", bucket, path);\n    }\n  }","sourceCodeStart":55,"sourceCodeEnd":91,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java#L55-L91","documentation":"GoogleDataSegmentKiller.kill deletes a segment's index.json and descriptor.json objects from Google Cloud Storage when a segment is killed. Any Google Cloud Storage client StorageException (permissions, missing bucket, API errors, timeouts) is rethrown as SegmentLoadingException carrying the segment ID and the GCS error message.","triggerScenarios":"Calling kill(segment) when the GCS API raises StorageException while listing or deleting the segment's objects: e.g. 403 on the service account, 404 for a missing bucket, rate limiting (429), or network timeouts during the delete.","commonSituations":"Service account lacking storage.objects.delete permission on the bucket; wrong druid.google.bucket/prefix config; segment already deleted by a concurrent kill task; GCS quota exhaustion or outage.","solutions":["Grant the service account storage.objects.delete (and list) roles on the bucket","Verify bucket/prefix configuration matches where segments were actually pushed","Check whether the segment was already killed (404) and treat as success in retry logic","Retry on transient StorageException codes (429, 5xx); the killer supports retry"],"exampleFix":"// before\nkiller.kill(segment); // throws on transient GCS errors\n// after\ntry {\n  killer.kill(segment);\n} catch (SegmentLoadingException e) {\n  if (isRetryable(e)) retryWithBackoff(() -> killer.kill(segment));\n  else throw e;\n}","handlingStrategy":"retry","validationCode":"// Check IAM before kill: gcloud storage buckets get-iam-policy gs://bucket --format=json","typeGuard":null,"tryCatchPattern":"try { killer.kill(segment); } catch (SegmentLoadingException e) { if (retryable(e)) backoffRetry(); else throw e; }","preventionTips":["Grant storage.objects.delete + list to the Druid service account","Confirm bucket/prefix config matches deep storage","Handle already-deleted segments as success","Retry transient 429/5xx StorageExceptions"],"tags":["google-cloud-storage","segment-kill","deep-storage","apache-druid"],"backgroundTag":"gcs-delete-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}