apache/beam · error · IllegalStateException

Logging stream terminated unexpectedly with success before i

Error message

Logging stream terminated unexpectedly with success before it was closed by the client.

What it means

BeamFnLoggingClient streams log messages from the SDK harness to the runner over a gRPC stream. This IllegalStateException is thrown by drainQueueToStream when the server half of the logging stream completes with success before the client closed its own end of the stream, which the Fn API logging protocol forbids. It indicates the runner terminated the log stream early, so subsequent log messages could not be delivered.

Source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java:257

            }
            continue;
          }

          // Batch together as many log messages as possible that are held within the buffer
          BeamFnApi.LogEntry.List.Builder builder =
              BeamFnApi.LogEntry.List.newBuilder().addLogEntries(logEntry);
          bufferedLogEntries.drainTo(additionalLogEntries);
          builder.addAllLogEntries(additionalLogEntries);
          outboundObserver.onNext(builder.build());
          additionalLogEntries.clear();
        }
        if (inboundObserverCompletion.isDone()) {
          try {
            // If the inbound observer failed with an exception, get() will throw an
            // ExecutionException.
            inboundObserverCompletion.get();
            // Otherwise it is an error for the server to close the stream before we closed our end.
            throw new IllegalStateException(
                "Logging stream terminated unexpectedly with success before it was closed by the client.");
          } catch (ExecutionException e) {
            throw new IllegalStateException(
                "Logging stream terminated unexpectedly before it was closed by the client with error: "
                    + e.getCause());
          } catch (InterruptedException e) {
            // Should never happen because of the isDone check.
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
          }
        }
      } catch (OutOfMemoryError oom) {
        throw oom;
      } catch (Throwable t) {
        thrown = t;
        throw new RuntimeException(t);
      } finally {
        if (thrown == null) {

View on GitHub (pinned to 12126d8942)

Solutions

  1. Check the runner version/health — this usually indicates the runner closed the logging stream prematurely; upgrade runner and Beam SDK to matching versions.
  2. Inspect for errors/aborts around the same instruction that would cause the runner to end the stream early (bundle failure, cancellation).
  3. Reduce excessive logging volume that keeps the queue draining long after the runner considers the stage done.
  4. Retry/re-run the pipeline; if reproducible, file an issue with runner logs.

Example fix

// before: relying on logging during/after stream close
LOG.info("stage finished after cleanup");
// after: guard logging so it stops once the stream is closing
if (!loggingClient.isClosed()) {
  LOG.info("stage finished after cleanup");
}
Defensive patterns

Strategy: try-catch

Try / catch

try {
  // pipeline work with logging
} catch (IllegalStateException e) {
  if (e.getMessage().contains("terminated unexpectedly")) {
    LOG.warn("Log stream closed early by runner; logs may be incomplete", e);
  } else { throw e; }
}

Prevention

When it happens

Trigger: The runner/server calls onCompleted() on the inbound logging observer while the client is still draining its log queue to the stream (client has not yet called close on its request observer).

Common situations: Runner shutting down a bundle or worker while the SDK harness still logs; runner bugs or aggressive timeouts closing the instruction logging stream early; pipeline finishing while deferred log messages remain queued.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/817c5af6f3768724. Report an issue: GitHub.