apache/druid · error · RuntimeException

Failed to upload [%s] to [%s]

Error message

Failed to upload [%s] to [%s]

What it means

GoogleTaskLogs.pushTaskFile uploads a task log/report/status file to GCS. Any non-IOException failure during upload (per-file, after the configured retry attempts) is rethrown as this RuntimeException naming the local file and the target GCS key. It means the push permanently failed for that file.

Source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java:113

      InputStreamContent mediaContent = new InputStreamContent("text/plain", fileStream);
      mediaContent.setLength(logFile.length());

      try {
        RetryUtils.retry(
            (RetryUtils.Task<Void>) () -> {
              storage.insert(config.getBucket(), taskKey, mediaContent, UPLOAD_BUFFER_SIZE);
              return null;
            },
            GoogleUtils::isRetryable,
            1,
            5
        );
      }
      catch (IOException e) {
        throw e;
      }
      catch (Exception e) {
        throw new RE(e, "Failed to upload [%s] to [%s]", logFile, taskKey);
      }
    }
  }

  @Override
  public Optional<InputStream> streamTaskLog(final String taskid, final long offset) throws IOException
  {
    final String taskKey = getTaskLogKey(taskid);
    return streamTaskFile(offset, taskKey);
  }

  @Override
  public Optional<InputStream> streamTaskReports(String taskid) throws IOException
  {
    final String taskKey = getTaskReportKey(taskid);
    return streamTaskFile(0, taskKey);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Read the cause chain (RE wraps the original exception) to find the real StorageException and its reason.
2. Verify the service account has roles/storage.objectCreator on the target bucket.
3. Confirm the bucket exists and google-cloud-storage API is enabled for the project.
4. Check GoogleTaskLogsConfig: bucket, prefix, and retry counts are sensible.
5. Because the exception is a RuntimeException, wrap the push call if task completion should tolerate log-upload failure.

Example fix

// before
taskLogs.pushTaskLog(taskId, logFile); // may blow up the task
// after
try {
  taskLogs.pushTaskLog(taskId, logFile);
} catch (RuntimeException e) {
  log.error(e, "Could not push task log to GCS; keeping local copy %s", logFile);
}

if (!logFile.exists()) {
  throw new IllegalStateException("Nothing to push: " + logFile);
}
// and confirm bucket is set in GoogleTaskLogsConfig

try { taskLogs.pushTaskLog(taskId, logFile); }
catch (RuntimeException e) {
  log.error(e, "GCS push failed for %s; local file retained", logFile);
}

When it happens

Trigger: Calling pushTaskLog/pushTaskReports/pushTaskStatus when GCS upload fails with something other than IOException — e.g. StorageException from auth failures, bucket absence, quota, or a null/bad config key — after retries are exhausted.

Common situations: Service account missing storage.objects.create; bucket does not exist or name typo; disabled GCS APIs on the project; network partitions during a long-running task finish; misconfigured maxAttempts/retry config.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failed to stream logs from: %s
• RuntimeException wrapping underlying cause
• Failed to upload [%s] to [%s]
• Failed to fetch google cloud storage object from bucket [%s]
• Failed to fetch google cloud storage object from bucket [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4d74d515d8ee526b. Report an issue: GitHub.