apache/beam · error · IllegalStateException

The accumulators cannot be merged:${e.getMessage()}

Error message

The accumulators cannot be merged:${e.getMessage()}

What it means

SketchFrequencies mergeAccumulators merges CountMinSketch instances via CountMinSketch.merge, which throws FrequencyMergeException when sketches were built with incompatible parameters (different depth/width/confidence). The code treats this as unreachable because all accumulators come from the same configured CountMinSketchFn, so it becomes an IllegalStateException indicating inconsistent accumulator state.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java:415

    @Override
    public Sketch<InputT> addInput(Sketch<InputT> accumulator, InputT element) {
      accumulator.add(element, inputCoder);

      return accumulator;
    }

    @Override
    public Sketch<InputT> mergeAccumulators(Iterable<Sketch<InputT>> accumulators) {
      Iterator<Sketch<InputT>> it = accumulators.iterator();
      Sketch<InputT> first = it.next();
      CountMinSketch mergedSketches = first.sketch();
      try {
        while (it.hasNext()) {
          mergedSketches = CountMinSketch.merge(mergedSketches, it.next().sketch());
        }
      } catch (FrequencyMergeException e) {
        // Should never happen because every instantiated accumulator are of the same type.
        throw new IllegalStateException("The accumulators cannot be merged:" + e.getMessage());
      }

      return Sketch.create(mergedSketches);
    }

    /** Output the whole structure so it can be queried, reused or stored easily. */
    @Override
    public Sketch<InputT> extractOutput(Sketch<InputT> accumulator) {
      return accumulator;
    }

    @Override
    public Coder<Sketch<InputT>> getAccumulatorCoder(CoderRegistry registry, Coder inputCoder) {
      return new CountMinSketchCoder<>();
    }

    @Override
    public void populateDisplayData(DisplayData.Builder builder) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Keep the withAccuracy(epsilon, confidence) configuration identical across pipeline restarts that resume stored state.
2. Pin the sketching extension (and stream-lib) version so serialized sketches remain compatible.
3. Discard old accumulator state and restart the job if the sketch configuration changed.
4. If reproducible with consistent configuration, report it as a Beam bug — this path should be unreachable.

Example fix

// before
.withAccuracy(0.05, 0.95) // changed after checkpoint was written with 0.01/0.999
// after
.withAccuracy(0.01, 0.999) // keep parameters stable across resuming runs, or start with fresh state

try { merged = CountMinSketch.merge(merged, next); } catch (FrequencyMergeException e) { /* restart with fresh state; should be unreachable */ }

When it happens

Trigger: Merging accumulators whose sketch parameters differ — practically only when accumulator state was persisted by a pipeline run using a different withAccuracy configuration or a different sketching-extension version and then merged in a resumed run.

Common situations: Streaming pipeline state upgraded between runs where epsilon/confidence changed; mixing checkpoints from jobs with different SketchFrequencies settings; dependency version drift of the sketching extension or stream-lib.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• The accumulators cannot be merged: ${e.getMessage()}
• Coder must be deterministic to perform this sketch.${e.getMe
• The relative error must be positive
• The confidence must be between 0 and 1
• Coder must be deterministic to perform this sketch.${e.getMe

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e084a2d637bac69b. Report an issue: GitHub.