apache/beam · error · UnsupportedOperationException

Does not support DISTINCT

Error message

Does not support %s DISTINCT

What it means

AggregationCombineFnAdapter.createCombineFn rejects AggregateCalls marked DISTINCT because the built-in CombineFn implementations cannot deduplicate inputs before aggregating. Beam SQL throws UnsupportedOperationException at query translation time.

Solutions

1. Rewrite the query using a subquery with GROUP BY on the distinct column before aggregating (e.g. SELECT COUNT(*) FROM (SELECT user GROUP BY user))
2. Use COUNT(DISTINCT x) only on versions/configurations that support it (HLL/UXTS variants)
3. Enable Beam SQL DISTINCT support flags or newer Beam version if available
4. Implement a custom UDAF with deduplication built in

Example fix

// before
SELECT COUNT(DISTINCT user_id) FROM events;
// after
SELECT COUNT(user_id) FROM (SELECT user_id FROM events GROUP BY user_id);

// Reject DISTINCT aggregates before submitting the query
if (sql.toUpperCase().matches(".*\\b(COUNT|SUM|AVG)\\s*\\(\\s*DISTINCT\\b.*")) {
  throw new IllegalArgumentException("Beam SQL does not support DISTINCT aggregates; rewrite query");
}

try {
  return AggregationCombineFnAdapter.createCombineFn(call, field, name);
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("DISTINCT")) {
    throw new IllegalArgumentException("Rewrite query to deduplicate via GROUP BY subquery", e);
  }
  throw e;
}

When it happens

Trigger: Running a SQL query like SELECT COUNT(DISTINCT x), SUM(DISTINCT x) FROM ... that translates into an AggregateCall with isDistinct()=true handled by a built-in CombineFn.

Common situations: Porting SQL from other engines (BigQuery/Calcite) that support DISTINCT aggregates; analytics queries counting unique users.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Beam SQL cannot convert Timestamp values with…
• buildIOWriter unsupported!
• CalcFn failed to evaluate
• Cannot create a filter for an unsupported node
• Could not compile CalcFn

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c7ce21fa86cc185e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java:145

    }

    @Override
    public Coder<Row> getAccumulatorCoder(CoderRegistry registry, Coder<Row> inputCoder)
        throws CannotProvideCoderException {
      return SchemaCoder.of(EMPTY_SCHEMA);
    }

    @Override
    public Coder<Row> getDefaultOutputCoder(CoderRegistry registry, Coder<Row> inputCoder) {
      return SchemaCoder.of(EMPTY_SCHEMA);
    }
  }

  /** Creates either a UDAF or a built-in {@link CombineFn}. */
  public static CombineFn<?, ?, ?> createCombineFn(
      AggregateCall call, Schema.Field field, String functionName) {
    if (call.isDistinct()) {
      throw new UnsupportedOperationException(
          "Does not support " + call.getAggregation().getName() + " DISTINCT");
    }

    CombineFn combineFn;
    if (call.getAggregation() instanceof SqlUserDefinedAggFunction) {
      combineFn = getUdafCombineFn(call);
    } else {
      combineFn = BeamBuiltinAggregations.create(functionName, field.getType());
    }
    if (call.getArgList().isEmpty()) {
      return new SingleInputCombiner(combineFn);
    } else if (call.getArgList().size() == 1) {
      return new SingleInputCombiner(combineFn);
    } else {
      return new MultiInputCombiner(combineFn);
    }
  }

View on GitHub (pinned to 12126d8942)