apache/druid · error · SegmentLoadingException

Couldn't kill segment

Error message

Couldn't kill segment[%s]: [%s]

What it means

GoogleDataSegmentKiller.kill deletes a segment's index.json and descriptor.json objects from Google Cloud Storage when a segment is killed. Any Google Cloud Storage client StorageException (permissions, missing bucket, API errors, timeouts) is rethrown as SegmentLoadingException carrying the segment ID and the GCS error message.

Solutions

1. Grant the service account storage.objects.delete (and list) roles on the bucket
2. Verify bucket/prefix configuration matches where segments were actually pushed
3. Check whether the segment was already killed (404) and treat as success in retry logic
4. Retry on transient StorageException codes (429, 5xx); the killer supports retry

Example fix

// before
killer.kill(segment); // throws on transient GCS errors
// after
try {
  killer.kill(segment);
} catch (SegmentLoadingException e) {
  if (isRetryable(e)) retryWithBackoff(() -> killer.kill(segment));
  else throw e;
}

// Check IAM before kill: gcloud storage buckets get-iam-policy gs://bucket --format=json

try { killer.kill(segment); } catch (SegmentLoadingException e) { if (retryable(e)) backoffRetry(); else throw e; }

When it happens

Trigger: Calling kill(segment) when the GCS API raises StorageException while listing or deleting the segment's objects: e.g. 403 on the service account, 404 for a missing bucket, rate limiting (429), or network timeouts during the delete.

Common situations: Service account lacking storage.objects.delete permission on the bucket; wrong druid.google.bucket/prefix config; segment already deleted by a concurrent kill task; GCS quota exhaustion or outage.

Related errors

• Cannot delete all segment files from Google Deep Storage…
• Failed to delete google cloud storage object from bucket
• Failed to fetch google cloud storage object from bucket
• Failed to upload [ ] to [ ]
• Invalid input file path

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a9a402db98174688. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java:73

  @Override
  public void kill(DataSegment segment) throws SegmentLoadingException
  {
    LOG.info("Killing segment [%s]", segment.getId());

    Map<String, Object> loadSpec = segment.getLoadSpec();
    final String bucket = MapUtils.getString(loadSpec, "bucket");
    final String indexPath = MapUtils.getString(loadSpec, "path");
    final String descriptorPath = DataSegmentKiller.descriptorPath(indexPath);

    try {
      deleteIfPresent(bucket, indexPath);
      // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored
      // anymore, but we still delete them if exists.
      deleteIfPresent(bucket, descriptorPath);
    }
    catch (StorageException e) {
      throw new SegmentLoadingException(e, "Couldn't kill segment[%s]: [%s]", segment.getId(), e.getMessage());
    }
  }

  private void deleteIfPresent(String bucket, String path)
  {
    try {
      GoogleUtils.retryGoogleCloudStorageOperation(() -> {
        storage.delete(bucket, path);
        return null;
      });
    }
    catch (StorageException e) {
      throw e;
    }
    catch (Exception e) {
      throw new RE(e, "Failed to delete google cloud storage object from bucket [%s] and path [%s].", bucket, path);
    }
  }

View on GitHub (pinned to 9b90983fd2)