apache/beam · error · RuntimeException

CalcFn failed to evaluate: ${processElementBlock}

Error message

CalcFn failed to evaluate: ${processElementBlock}

What it means

At runtime the compiled CalcFn evaluates each row; when evaluation throws and the pipeline is not configured with an error-output (error_fn/exception handling), the cause is rethrown as a RuntimeException naming the generated processElement block. It surfaces user-code failures inside SQL expressions such as NPEs, arithmetic errors, or UDF exceptions.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java:350

        OutputReceiver<Row> outputReceiver,
        MultiOutputReceiver multiOutputReceiver) {
      assert se != null;
      try {
        Object[] v = (Object[]) se.evaluate(new Object[] {row, CONTEXT_INSTANCE});
        if (v != null) {
          final Row output = toBeamRow(Arrays.asList(v), outputSchema, verifyRowValues);
          outputReceiver.output(output);
        }

      } catch (InvocationTargetException e) {
        if (collectErrors) {
          Schema schema = BeamSqlRelUtils.getErrorRowSchema(row.getSchema());
          Row errorRow =
              toBeamRow(Arrays.asList(row.getValues(), e.getCause().getMessage()), schema, true);
          LOG.error("CalcFn failed to evaluate: {}", processElementBlock, e.getCause());
          multiOutputReceiver.get(errors).output(errorRow);
        } else {
          throw new RuntimeException(
              "CalcFn failed to evaluate: " + processElementBlock, e.getCause());
        }
      }
    }
  }

  private static List<String> getJarPaths(RexProgram program) {
    ImmutableList.Builder<String> jarPaths = new ImmutableList.Builder<>();
    for (RexNode node : program.getExprList()) {
      if (node instanceof RexCall) {
        SqlOperator op = ((RexCall) node).op;
        if (op instanceof SqlUserDefinedFunction) {
          Function function = ((SqlUserDefinedFunction) op).function;
          if (function instanceof ScalarFunctionImpl) {
            String jarPath = ((ScalarFunctionImpl) function).getJarPath();
            if (!jarPath.isEmpty()) {
              jarPaths.add(jarPath);
            }

View on GitHub (pinned to 12126d8942)

Solutions

1. Read e.getCause() in the exception — it holds the real failure (NPE, UDF exception, etc.) and the failing row
2. Use Beam SQL's error-tolerant mode: configure the multi-output/errors collection (error row schema) so bad rows are routed instead of failing the pipeline
3. Null-guard expressions in the query (e.g. COALESCE/IFNULL) and in UDFs
4. Fix the offending input data or make the UDF defensive

Example fix

// before
SELECT quantity / total AS ratio FROM orders;
// after
SELECT IFNULL(quantity / NULLIF(total, 0), 0) AS ratio FROM orders;

// Pre-validate rows: reject rows with nulls in columns used by the query
boolean hasRequiredValues(Row r, List<String> cols) {
  return cols.stream().noneMatch(c -> r.getValue(c) == null);
}

try {
  rowOut = se.eval(...);
} catch (Throwable e) {
  // inspect e.getCause(); route row to errors output instead of failing pipeline
}

When it happens

Trigger: A row hits the Calc operator and the generated expression throws — null operand for a non-null-safe operator, division by zero, a UDF throwing, or a type cast failing at runtime for a specific record (other records may succeed).

Common situations: Null field values feeding SQL functions; dirty data in streaming sources; UDF that assumes non-null or well-formed input; numeric overflow in arithmetic expressions.

Understand the failure class

Background: "query failed", "%w: SQL error" — wrapped database query errors in Go libraries explained — this error's family across 3 libraries.

Related errors

• Failed to load user-provided jar(s).
• returnLength cannot be 0 or pattern cannot be empty.
• ${config}
• ${fieldType}
• ${diagnostics}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ab691bad9c77cc85. Report an issue: GitHub.