apache/beam · error · IllegalStateException

The accumulators cannot be merged: ${e.getMessage()}

Error message

The accumulators cannot be merged: ${e.getMessage()}

What it means

ApproximateDistinctFn.mergeAccumulators merges HyperLogLogPlus accumulators via addAll, which can throw CardinalityMergeException when sketches have incompatible parameters (different p/sp registers). The code comments this 'should never happen' because only HyperLogLogPlus accumulators are instantiated, so the IllegalStateException signals an internal invariant violation.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java:478

      }
      return acc;
    }

    /** Output the whole structure so it can be queried, reused or stored easily. */
    @Override
    public HyperLogLogPlus extractOutput(HyperLogLogPlus accumulator) {
      return accumulator;
    }

    @Override
    public HyperLogLogPlus mergeAccumulators(Iterable<HyperLogLogPlus> accumulators) {
      HyperLogLogPlus mergedAccum = createAccumulator();
      for (HyperLogLogPlus accum : accumulators) {
        try {
          mergedAccum.addAll(accum);
        } catch (CardinalityMergeException e) {
          // Should never happen because only HyperLogLogPlus accumulators are instantiated.
          throw new IllegalStateException(
              "The accumulators cannot be merged: " + e.getMessage(), e);
        }
      }
      return mergedAccum;
    }

    @Override
    public void populateDisplayData(DisplayData.Builder builder) {
      super.populateDisplayData(builder);
      builder
          .add(DisplayData.item("p", p).withLabel("precision"))
          .add(DisplayData.item("sp", sp).withLabel("sparse representation precision"));
    }
  }

  /** Coder for {@link HyperLogLogPlus} class. */
  public static class HyperLogLogPlusCoder extends CustomCoder<HyperLogLogPlus> {

View on GitHub (pinned to 12126d8942)

Solutions

1. Do not mix accumulator state from pipeline versions with different ApproximateDistinct configuration; restart with fresh state.
2. Keep the sketching extension version consistent between save and restore of streaming state.
3. If reproducible, file a Beam issue — this path is expected to be unreachable.

try { mergedAccum.addAll(accum); } catch (CardinalityMergeException e) { /* restart with fresh state; should be unreachable */ }

When it happens

Trigger: Merging accumulators that were created with different precision/bias parameters — practically only when accumulator state was deserialized from data written with a different configuration (changed p2 parameter or sketch version between pipeline runs).

Common situations: Resuming/updating a streaming pipeline whose stored accumulator state was produced by a different version of the sketching extension or with different HLL++ parameters; mixing sketch outputs from jobs with different configs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Coder must be deterministic to perform this sketch.${e.getMe
• The accumulators cannot be merged:${e.getMessage()}
• The input value cannot be encoded: ${e.getMessage()}
• cannot encode a null HyperLogLogPlus sketch
• Coder must be deterministic to perform this sketch.${e.getMe

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/65be091e2427a7d1. Report an issue: GitHub.