apache/beam · error · RuntimeException

Error starting background log thread

Error message

Error starting background log thread 

What it means

BeamFnLoggingClient starts a background thread that consumes buffered log messages and forwards them to the logging service over gRPC. If that thread fails to start or dies during startup, the CompletableFuture completes exceptionally and the constructor rethrows it as RuntimeException('Error starting background log thread ...').

Source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java:175

                // Logging which occurs in this thread will attempt to publish log entries into the
                // above handler which should never block if the queue is full otherwise
                // this thread will get stuck.
                streamWriter.drainQueueToStream(bufferedLogEntries);
              } finally {
                restoreLoggers();
                // Now that loggers are restored, do a final flush of any buffered logs
                // in case they help with understanding above failures.
                flushFinalLogs();
              }
              return COMPLETED;
            },
            executorService);
    try {
      // Wait for the thread to be running and log handlers installed or an error with the thread
      // that is supposed to be consuming logs.
      CompletableFuture.anyOf(this.bufferedLogConsumer, started).get();
    } catch (ExecutionException e) {
      throw new RuntimeException("Error starting background log thread " + e.getCause());
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new RuntimeException(e);
    }
  }

  @RequiresNonNull("logRecordHandler")
  @RequiresNonNull("configuredLoggers")
  private void installLogging(
      @UnderInitialization BeamFnLoggingClient this, SdkHarnessOptions options) {
    // Reset the global log manager, get the root logger and remove the default log handlers.
    LogManager logManager = LogManager.getLogManager();
    logManager.reset();
    Logger rootLogger = logManager.getLogger(ROOT_LOGGER_NAME);
    for (Handler handler : rootLogger.getHandlers()) {
      rootLogger.removeHandler(handler);
    }
    // configure loggers from default sdk harness log level and log level overrides

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the wrapped cause (e.getCause()) in the message to find the root failure (gRPC channel, thread start, etc.).
2. Verify the logging ApiServiceDescriptor endpoint is correct and reachable from the SDK worker.
3. Check executor/thread availability (ulimit -u, container thread limits) if the cause is a thread-start failure.
4. Upgrade/match runner and harness versions to rule out protocol incompatibilities, then retry the worker.

// pre-check endpoint reachability: assert !loggingEndpoint.isEmpty();

try { new BeamFnLoggingClient(...); } catch (RuntimeException e) { LOG.error("Log thread startup failed", e.getCause()); /* reconfigure endpoint or fail worker */ throw e; }

When it happens

Trigger: The background log-consuming thread throws before/while installing log handlers (e.g. gRPC channel setup failure, uncaught exception in the consumer runnable), causing ExecutionException on the anyOf(...).get() future.

Common situations: Logging service endpoint unreachable or misconfigured (bad env var like worker logging service address); thread factory/executor unable to create threads (resource exhaustion, thread limits); serialization failures of log records at startup.

Related errors

• Unsupported log level '%s' requested for %s. Must be one of
• Unknown log level ${level}. Valid log levels are ${validLeve
• Timers are unsupported because the ProcessBundleRequest %s d
• Unknown URN %s
• Unable to construct %s counter for PTransform {id=%s, name=%

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3b83da703709a176. Report an issue: GitHub.