apache/beam · error · IllegalStateException

The accumulators cannot be merged

Error message

The accumulators cannot be merged: ${e.getMessage()}

What it means

ApproximateDistinctFn.mergeAccumulators merges HyperLogLogPlus accumulators via addAll, which can throw CardinalityMergeException when sketches have incompatible parameters (different p/sp registers). The code comments this 'should never happen' because only HyperLogLogPlus accumulators are instantiated, so the IllegalStateException signals an internal invariant violation.

Solutions

1. Do not mix accumulator state from pipeline versions with different ApproximateDistinct configuration; restart with fresh state.
2. Keep the sketching extension version consistent between save and restore of streaming state.
3. If reproducible, file a Beam issue — this path is expected to be unreachable.

try { mergedAccum.addAll(accum); } catch (CardinalityMergeException e) { /* restart with fresh state; should be unreachable */ }

When it happens

Trigger: Merging accumulators that were created with different precision/bias parameters — practically only when accumulator state was deserialized from data written with a different configuration (changed p2 parameter or sketch version between pipeline runs).

Common situations: Resuming/updating a streaming pipeline whose stored accumulator state was produced by a different version of the sketching extension or with different HLL++ parameters; mixing sketch outputs from jobs with different configs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Coder must be deterministic to perform this sketch.
• The accumulators cannot be merged
• cannot encode a null HyperLogLogPlus sketch
• Coder must be deterministic to perform this sketch.
• The confidence must be between 0 and 1

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/65be091e2427a7d1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java:478

      }
      return acc;
    }

    /** Output the whole structure so it can be queried, reused or stored easily. */
    @Override
    public HyperLogLogPlus extractOutput(HyperLogLogPlus accumulator) {
      return accumulator;
    }

    @Override
    public HyperLogLogPlus mergeAccumulators(Iterable<HyperLogLogPlus> accumulators) {
      HyperLogLogPlus mergedAccum = createAccumulator();
      for (HyperLogLogPlus accum : accumulators) {
        try {
          mergedAccum.addAll(accum);
        } catch (CardinalityMergeException e) {
          // Should never happen because only HyperLogLogPlus accumulators are instantiated.
          throw new IllegalStateException(
              "The accumulators cannot be merged: " + e.getMessage(), e);
        }
      }
      return mergedAccum;
    }

    @Override
    public void populateDisplayData(DisplayData.Builder builder) {
      super.populateDisplayData(builder);
      builder
          .add(DisplayData.item("p", p).withLabel("precision"))
          .add(DisplayData.item("sp", sp).withLabel("sparse representation precision"));
    }
  }

  /** Coder for {@link HyperLogLogPlus} class. */
  public static class HyperLogLogPlusCoder extends CustomCoder<HyperLogLogPlus> {

View on GitHub (pinned to 12126d8942)