{"record":{"id":"774edfe75bc99039","repo":"apache/beam","slug":"operation-ongoing-in-bundle-for-at-least-without-outputting","errorCode":null,"errorMessage":"Operation ongoing in bundle {} for at least {} without outputting or completing (stack trace unable to be generated).","messagePattern":"Operation ongoing in bundle (.+?) for at least (.+?) without outputting or completing \\(stack trace unable to be generated\\)\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ExecutionStateSampler.java","lineNumber":453,"sourceCode":"                    currentExecutionState.ptransformUniqueName,\n                    currentExecutionState.stateName,\n                    DURATION_FORMATTER.print(\n                        Duration.millis(userSpecifiedLullTimeMsForRestart).toPeriod()),\n                    Joiner.on(\"\\n  at \").join(thread.getStackTrace()));\n          }\n          return Optional.of(timeoutMessage);\n        }\n\n        if (lullTimeMs > MAX_LULL_TIME_MS) {\n          if (lullTimeMs < lastLullReport // This must be a new report.\n              || lullTimeMs > 1.2 * lastLullReport // Exponential backoff.\n              || lullTimeMs\n                  > MAX_LULL_TIME_MS + lastLullReport // At least once every MAX_LULL_TIME_MS.\n          ) {\n            lastLullReport = lullTimeMs;\n            Thread thread = trackedThread.get();\n            if (thread == null) {\n              LOG.warn(\n                  \"Operation ongoing in bundle {} for at least {} without outputting \"\n                      + \"or completing (stack trace unable to be generated).\",\n                  processBundleId.get(),\n                  DURATION_FORMATTER.print(Duration.millis(lullTimeMs).toPeriod()));\n            } else if (currentExecutionState == null) {\n              LOG.warn(\n                  \"Operation ongoing in bundle {} for at least {} without outputting \"\n                      + \"or completing:\\n  at {}\",\n                  processBundleId.get(),\n                  DURATION_FORMATTER.print(Duration.millis(lullTimeMs).toPeriod()),\n                  Joiner.on(\"\\n  at \").join(thread.getStackTrace()));\n            } else {\n              LOG.warn(\n                  \"Operation ongoing in bundle {} for PTransform{{id={}, name={}, state={}}} \"\n                      + \"for at least {} without outputting or completing:\\n  at {}\",\n                  processBundleId.get(),\n                  currentExecutionState.ptransformId,\n                  currentExecutionState.ptransformUniqueName,","sourceCodeStart":435,"sourceCodeEnd":471,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ExecutionStateSampler.java#L435-L471","documentation":"The Java Fn Harness ExecutionStateSampler logs a lull warning when a bundle's current operation has been running longer than MAX_LULL_TIME_MS without producing output or completing and no stack trace can be generated because the tracked thread reference is null. It helps diagnose stuck/hung processing stages.","triggerScenarios":"takeSample observes lullTimeMs > maxLullTime while the trackedThread WeakReference has been cleared (thread already exited) — typically a very long-running or stalled ProcessBundle operation.","commonSituations":"Slow external I/O in a DoFn (database, HTTP) with no timeouts; huge state/side-input reads; deadlocked or finished threads whose sampling state wasn't cleaned up; large elements causing long user-code execution.","solutions":["Inspect which stage/bundle is lulling and add timeouts or metrics in the user DoFn's I/O calls.","Increase the sampler threshold only for legitimately slow operations: PipelineOptions.setMaxLullTimeMs or the equivalent flag.","Use thread dumps / JFR on the worker to find the stuck code; look for blocking external calls.","Check for resource contention (connection pools exhausted, slow storage) on the worker host."],"exampleFix":"// before\ntry (Connection c = pool.getConnection()) { /* no timeout; can lull forever */ }\n// after\npool.setMaxWait(Duration.ofSeconds(30));\nstmt.setQueryTimeout(60); // bounds operation duration, avoids lull warnings\n","handlingStrategy":"validation","validationCode":"// before submitting, bound all blocking I/O in DoFns\nstmt.setQueryTimeout(60);\nhttpClient.timeout(Duration.ofSeconds(30));\n// and optionally raise the sampler threshold for slow stages\noptions.setMaxLullTimeMs(Duration.minutes(10).getMillis());","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Add timeouts to every external I/O call inside DoFns","Monitor lull warnings and correlate with thread dumps/JFR","Ensure the thread performing the operation stays alive so stack traces can be sampled","Tune maxLullTimeMs per stage for legitimately slow operations"],"tags":["java","harness","performance","stuck-thread","lull"],"backgroundTag":"operation-lull-detected","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}