apache/beam · error · java.lang.RuntimeException

Pipeline failed for unknown reason

Error message

Pipeline failed for unknown reason

What it means

BeamEnumerableConverter.limitRun executes a Beam SQL pipeline (built from the BeamRelNode) and polls the pipeline result state every second until it finishes, while collecting rows up to a LIMIT. If the pipeline reaches a terminal FAILED state, the original underlying exception is discarded and this generic RuntimeException("Pipeline failed for unknown reason") is thrown instead, hiding the real cause from the caller.

Solutions

1. Run the pipeline with DEBUG logging for org.apache.beam to find the root-cause exception printed before this error
2. Inspect the runner's job logs (Flink/Spark/Dataflow UI or DirectRunner console) for the original failure
3. Call the query without LIMIT (collectRows path) or with a plain PipelineResult to surface the underlying exception
4. Verify input table configs (beamSql table types, connection params) and that all UDFs/POJOs are Serializable
5. Pin/upgrade Beam and runner versions to compatible releases to rule out known converter bugs

Example fix

// before (library code loses the cause)
if (PipelineResult.State.FAILED.equals(state)) {
  throw new RuntimeException("Pipeline failed for unknown reason");
}
// after
if (PipelineResult.State.FAILED.equals(state)) {
  Throwable cause = result instanceof CompleteFutureResult
      ? null : null; // or capture error callback
  throw new RuntimeException("Pipeline failed for unknown reason", cause);
}

// verify pipeline completes and inspect state before relying on the limited rows
PipelineResult result = pipeline.run();
PipelineResult.State state = result.waitUntilFinish();
if (state != PipelineResult.State.DONE) {
  throw new IllegalStateException("Beam SQL pipeline did not finish: " + state);
}

boolean pipelineSucceeded(PipelineResult r) {
  return r != null && PipelineResult.State.DONE.equals(r.waitUntilFinish());
}

try {
  Enumerable<Object> rows = limitRun(options, node, limit);
} catch (RuntimeException e) {
  if ("Pipeline failed for unknown reason".equals(e.getMessage())) {
    // consult runner logs for the suppressed root cause; wrap or retry
  } else throw e;
}

When it happens

Trigger: Calling LIMIT-style SQL execution (BeamSqlStringUtils/BeamSqlEnv query paths that route through limitCollect -> limitRun) on a pipeline whose job fails during execution — e.g. a bad source/sink configuration, an exception inside a user DoFn, quota/resource errors on the runner, or invalid input data causing a worker crash.

Common situations: Developers running Beam SQL interactively via calcite CLI or sqlline, or unit-testing BeamSql queries with a DirectRunner/SparkRunner/FlinkRunner, where the pipeline aborts (missing module, bad test table config, serialization errors of the BeamRelNode or UDF classes) and the real stack trace is swallowed.

Related errors

• A list of URNs for overriding transforms was provided but…
• A cannot be expanded
• A transform cannot be initiated using the provided config…
• AVRO schema doesn't match row schema. Row schema
• BigQuery data contained value

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/721cb4d6d2787f21. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java:195

      PipelineOptions options,
      BeamRelNode node,
      DoFn<Row, Void> doFn,
      Queue<Row> values,
      int limitCount) {
    options.as(DirectOptions.class).setBlockOnRun(false);
    Pipeline pipeline = Pipeline.create(options);
    PCollection<Row> resultCollection = BeamSqlRelUtils.toPCollection(pipeline, node);
    resultCollection.apply(ParDo.of(doFn));

    PipelineResult result = pipeline.run();

    State state;
    while (true) {
      // Check pipeline state in every second
      state = result.waitUntilFinish(Duration.standardSeconds(1));
      if (state != null && state.isTerminal()) {
        if (PipelineResult.State.FAILED.equals(state)) {
          throw new RuntimeException("Pipeline failed for unknown reason");
        }
        break;
      }

      try {
        if (values.size() >= limitCount) {
          result.cancel();
          break;
        }
      } catch (IOException e) {
        LOG.warn("{}", e.toString());
        break;
      }
    }

    return result;
  }

View on GitHub (pinned to 12126d8942)