Operation ongoing in bundle

Error message

Operation ongoing in bundle {} for at least {} without outputting or completing (stack trace unable to be generated).

What it means

The Java Fn Harness ExecutionStateSampler logs a lull warning when a bundle's current operation has been running longer than MAX_LULL_TIME_MS without producing output or completing and no stack trace can be generated because the tracked thread reference is null. It helps diagnose stuck/hung processing stages.

Solutions

Inspect which stage/bundle is lulling and add timeouts or metrics in the user DoFn's I/O calls.
Increase the sampler threshold only for legitimately slow operations: PipelineOptions.setMaxLullTimeMs or the equivalent flag.
Use thread dumps / JFR on the worker to find the stuck code; look for blocking external calls.
Check for resource contention (connection pools exhausted, slow storage) on the worker host.

Example fix

// before
try (Connection c = pool.getConnection()) { /* no timeout; can lull forever */ }
// after
pool.setMaxWait(Duration.ofSeconds(30));
stmt.setQueryTimeout(60); // bounds operation duration, avoids lull warnings

Defensive patterns

Strategy: validation

Validate before calling

// before submitting, bound all blocking I/O in DoFns
stmt.setQueryTimeout(60);
httpClient.timeout(Duration.ofSeconds(30));
// and optionally raise the sampler threshold for slow stages
options.setMaxLullTimeMs(Duration.minutes(10).getMillis());

Prevention

Add timeouts to every external I/O call inside DoFns
Monitor lull warnings and correlate with thread dumps/JFR
Ensure the thread performing the operation stays alive so stack traces can be sampled
Tune maxLullTimeMs per stage for legitimately slow operations

When it happens

Trigger: takeSample observes lullTimeMs > maxLullTime while the trackedThread WeakReference has been cleared (thread already exited) — typically a very long-running or stalled ProcessBundle operation.

Common situations: Slow external I/O in a DoFn (database, HTTP) with no timeouts; huge state/side-input reads; deadlocked or finished threads whose sampling state wasn't cleaned up; large elements causing long user-code execution.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/774edfe75bc99039. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ExecutionStateSampler.java:453

                    currentExecutionState.ptransformUniqueName,
                    currentExecutionState.stateName,
                    DURATION_FORMATTER.print(
                        Duration.millis(userSpecifiedLullTimeMsForRestart).toPeriod()),
                    Joiner.on("\n  at ").join(thread.getStackTrace()));
          }
          return Optional.of(timeoutMessage);
        }

        if (lullTimeMs > MAX_LULL_TIME_MS) {
          if (lullTimeMs < lastLullReport // This must be a new report.
              || lullTimeMs > 1.2 * lastLullReport // Exponential backoff.
              || lullTimeMs
                  > MAX_LULL_TIME_MS + lastLullReport // At least once every MAX_LULL_TIME_MS.
          ) {
            lastLullReport = lullTimeMs;
            Thread thread = trackedThread.get();
            if (thread == null) {
              LOG.warn(
                  "Operation ongoing in bundle {} for at least {} without outputting "
                      + "or completing (stack trace unable to be generated).",
                  processBundleId.get(),
                  DURATION_FORMATTER.print(Duration.millis(lullTimeMs).toPeriod()));
            } else if (currentExecutionState == null) {
              LOG.warn(
                  "Operation ongoing in bundle {} for at least {} without outputting "
                      + "or completing:\n  at {}",
                  processBundleId.get(),
                  DURATION_FORMATTER.print(Duration.millis(lullTimeMs).toPeriod()),
                  Joiner.on("\n  at ").join(thread.getStackTrace()));
            } else {
              LOG.warn(
                  "Operation ongoing in bundle {} for PTransform{{id={}, name={}, state={}}} "
                      + "for at least {} without outputting or completing:\n  at {}",
                  processBundleId.get(),
                  currentExecutionState.ptransformId,
                  currentExecutionState.ptransformUniqueName,

View on GitHub (pinned to 12126d8942)