apache/beam · error · IllegalStateException

Logging stream terminated unexpectedly before it was closed

Error message

Logging stream terminated unexpectedly before it was closed by the client with error: 

What it means

BeamFnLoggingClient's drainQueueToStream waits on inboundObserverCompletion; if that future completes exceptionally, the server terminated the logging gRPC stream with an error before the client closed its end. The cause of the remote failure is appended to the message. This signals the log channel to the runner died mid-stream.

Source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java:260

          // Batch together as many log messages as possible that are held within the buffer
          BeamFnApi.LogEntry.List.Builder builder =
              BeamFnApi.LogEntry.List.newBuilder().addLogEntries(logEntry);
          bufferedLogEntries.drainTo(additionalLogEntries);
          builder.addAllLogEntries(additionalLogEntries);
          outboundObserver.onNext(builder.build());
          additionalLogEntries.clear();
        }
        if (inboundObserverCompletion.isDone()) {
          try {
            // If the inbound observer failed with an exception, get() will throw an
            // ExecutionException.
            inboundObserverCompletion.get();
            // Otherwise it is an error for the server to close the stream before we closed our end.
            throw new IllegalStateException(
                "Logging stream terminated unexpectedly with success before it was closed by the client.");
          } catch (ExecutionException e) {
            throw new IllegalStateException(
                "Logging stream terminated unexpectedly before it was closed by the client with error: "
                    + e.getCause());
          } catch (InterruptedException e) {
            // Should never happen because of the isDone check.
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
          }
        }
      } catch (OutOfMemoryError oom) {
        throw oom;
      } catch (Throwable t) {
        thrown = t;
        throw new RuntimeException(t);
      } finally {
        if (thrown == null) {
          outboundObserver.onCompleted();
        } else {
          outboundObserver.onError(thrown);

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the appended cause to identify the gRPC status and fix the underlying transport/runner failure.
2. Check network connectivity and timeouts between the SDK harness worker and the runner service.
3. Ensure the runner keeps the log stream open for the lifetime of the instruction; align Beam SDK and runner versions.
4. Reduce log flood that can trip flow-control or deadline limits on the stream.

try {
  // harness/runner operation
} catch (IllegalStateException e) {
  Throwable cause = e.getCause();
  if (cause instanceof StatusRuntimeException && ((StatusRuntimeException) cause).getStatus().getCode().isRetryable()) {
    // retry or reschedule the work
  } else { throw e; }
}

When it happens

Trigger: The gRPC server-side observer of the log stream fails with an exception (ExecutionException) while the client is still sending log records — e.g. transport error, CANCELLED/UNAVAILABLE status, or server-side handler crash.

Common situations: Network interruption between worker and runner; worker exceeding a deadline; runner rejecting the log stream (e.g. instruction no longer known); container memory pressure killing the stream.

Related errors

• Logging stream terminated unexpectedly with success before i
• chunk send failed
• failed to receive header
• failed to retrieve chunk for %v
• failed to send header for %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2231678a43bed569. Report an issue: GitHub.