apache/druid · error · IOException

Failed to stream logs from: %s

Error message

Failed to stream logs from: %s

What it means

GoogleTaskLogs.streamTaskFile opens a GCS object for streaming task logs/reports/status. Any failure to open or read the object is wrapped into an IOException with this message and the GCS taskKey. It indicates the requested log could not be streamed, most often because it does not exist.

Source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java:166

      try {
        final long start;

        if (offset > 0 && offset < length) {
          start = offset;
        } else if (offset < 0 && (-1 * offset) < length) {
          start = length + offset;
        } else {
          start = 0;
        }

        return Optional.of(new GoogleByteSource(storage, config.getBucket(), taskKey).openStream(start));
      }
      catch (Exception e) {
        throw new IOException(e);
      }
    }
    catch (IOException e) {
      throw new IOE(e, "Failed to stream logs from: %s", taskKey);
    }
  }

  private String getTaskLogKey(String taskid)
  {
    return config.getPrefix() + "/" + taskid.replace(':', '_');
  }

  private String getTaskReportKey(String taskid)
  {
    return config.getPrefix() + "/" + taskid.replace(':', '_') + ".report.json";
  }

  private String getTaskStatusKey(String taskid)
  {
    return config.getPrefix() + "/" + taskid.replace(':', '_') + ".status.json";
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Confirm the object exists at <prefix>/<taskId with ':' replaced by '_'> in the configured bucket.
2. If the task is still running, expect this failure and serve the local file instead (callers normally fall back).
3. Check GoogleTaskLogsConfig bucket/prefix against where the task actually pushed.
4. Grant the reading node's credentials storage.objects.get on the bucket.
5. Inspect the wrapped cause for StorageException codes (404 vs 403 vs 5xx) to pick retry vs fix.

Example fix

// before
Optional<InputStream> in = taskLogs.streamTaskLog(taskId, 0);
// after
Optional<InputStream> in = Optional.empty();
try {
  in = taskLogs.streamTaskLog(taskId, 0);
} catch (IOException e) {
  log.warn("No task log in GCS for %s, falling back to local file", taskId);
}

String key = config.getPrefix() + "/" + taskid.replace(':', '_');
boolean exists = storage.get(config.getBucket(), key) != null;

try { return Optional.of(stream.open()); }
catch (IOException e) {
  log.warn("Task log not streamable from GCS: %s", taskKey);
  return Optional.empty(); // serve local file instead
}

When it happens

Trigger: Calling streamTaskLog/streamTaskReports/streamTaskStatus for a task whose file is not in GCS (task still running, log never pushed, wrong prefix), or when opening the GCS read channel fails (permissions, transient client errors).

Common situations: Overlord console streaming logs of a still-running task before push; wrong prefix configured in GoogleTaskLogsConfig; deleted logs past retention; service account lost read access.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to fetch google cloud storage object from bucket [%s]
• Failed to fetch google cloud storage object from bucket [%s]
• Failed to upload [%s] to [%s]
• Failed to stream logs for task[%s] starting at offset[%d]
• Failed to read log for task: %s

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/c906e4c8ed6faae7. Report an issue: GitHub.