apache/druid · error · RuntimeException

Failed to upload [%s] to [%s]

Error message

Failed to upload [%s] to [%s]

What it means

GoogleDataSegmentPusher.insert uploads a segment's zipped index file to Google Cloud Storage. Non-IOException errors from the upload (GCS client StorageException for permission/quota/API failures) are wrapped in a RuntimeException naming the source file and destination path; IOExceptions are rethrown as-is per the method contract.

Source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentPusher.java:77

      throws IOException
  {
    log.debug("Inserting [%s] to [%s]", file, path);
    try {
      RetryUtils.retry(
          (RetryUtils.Task<Void>) () -> {
            storage.insert(config.getBucket(), path, new FileContent(contentType, file), null);
            return null;
          },
          GoogleUtils::isRetryable,
          1,
          5
      );
    }
    catch (IOException e) {
      throw e;
    }
    catch (Exception e) {
      throw new RE(e, "Failed to upload [%s] to [%s]", file, path);
    }
  }

  @Override
  public DataSegment push(final File indexFilesDir, final DataSegment segment, final boolean useUniquePath)
      throws IOException
  {
    log.debug("Uploading [%s] to Google.", indexFilesDir);
    final String storageDir = this.getStorageDir(segment, useUniquePath);
    return pushToPath(indexFilesDir, segment, storageDir);
  }

  @Override
  public DataSegment pushToPath(File indexFilesDir, DataSegment segment, String storageDirSuffix) throws IOException
  {
    final int version = SegmentUtils.getVersionFromDir(indexFilesDir);
    File indexFile = null;

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Grant the service account storage.objects.create/roles-storage-objectAdmin on the bucket
2. Verify druid.storage.bucket (and prefix) point to an existing, correct bucket
3. Retry on transient failures (429/5xx); publishing is retried by the task framework
4. Check the cause StorageException message for the exact GCS API error code

Example fix

// before
pusher.push(segmentDir, segment, true); // RE: Failed to upload ...
// after
// ensure in GCS IAM: roles/storage.objectCreator for the Druid service account on the bucket
pusher.push(segmentDir, segment, true);

// Pre-flight: verify bucket exists and account has create rights
// gsutil ls gs://druid-storage-bucket && gcloud storage buckets get-iam-policy gs://druid-storage-bucket

try { pusher.push(dir, segment, true); } catch (Exception e) { if (isRetryableStorageError(e)) backoffRetry(); else throw e; }

When it happens

Trigger: Calling insert (via push during segment publishing) when the GCS blob create/upload raises StorageException (403 insufficient permissions, 404 bucket missing, 429 quota, resumable upload failures) or other RuntimeExceptions from the client.

Common situations: Service account lacking storage.objects.create on the target bucket; bucket name typo or deleted bucket; GCS rate limits during heavy segment publish; segment size exceeding GCS limits.

Related errors

• Invalid input file path [%s]
• Couldn't kill segment[%s]: [%s]
• Failed to delete google cloud storage object from bucket [%s
• Cannot delete all segment files from Google Deep Storage sin
• Failed to fetch google cloud storage object from bucket [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ae2bff2ed8dc7c1b. Report an issue: GitHub.