apache/beam · error · IllegalStateException

The accumulators cannot be merged

Error message

The accumulators cannot be merged:${e.getMessage()}

What it means

SketchFrequencies mergeAccumulators merges CountMinSketch instances via CountMinSketch.merge, which throws FrequencyMergeException when sketches were built with incompatible parameters (different depth/width/confidence). The code treats this as unreachable because all accumulators come from the same configured CountMinSketchFn, so it becomes an IllegalStateException indicating inconsistent accumulator state.

Solutions

1. Keep the withAccuracy(epsilon, confidence) configuration identical across pipeline restarts that resume stored state.
2. Pin the sketching extension (and stream-lib) version so serialized sketches remain compatible.
3. Discard old accumulator state and restart the job if the sketch configuration changed.
4. If reproducible with consistent configuration, report it as a Beam bug — this path should be unreachable.

Example fix

// before
.withAccuracy(0.05, 0.95) // changed after checkpoint was written with 0.01/0.999
// after
.withAccuracy(0.01, 0.999) // keep parameters stable across resuming runs, or start with fresh state

try { merged = CountMinSketch.merge(merged, next); } catch (FrequencyMergeException e) { /* restart with fresh state; should be unreachable */ }

When it happens

Trigger: Merging accumulators whose sketch parameters differ — practically only when accumulator state was persisted by a pipeline run using a different withAccuracy configuration or a different sketching-extension version and then merged in a resumed run.

Common situations: Streaming pipeline state upgraded between runs where epsilon/confidence changed; mixing checkpoints from jobs with different SketchFrequencies settings; dependency version drift of the sketching extension or stream-lib.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Coder must be deterministic to perform this sketch.
• The accumulators cannot be merged
• The confidence must be between 0 and 1
• The relative error must be positive
• cannot encode a null HyperLogLogPlus sketch

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e084a2d637bac69b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java:415

    @Override
    public Sketch<InputT> addInput(Sketch<InputT> accumulator, InputT element) {
      accumulator.add(element, inputCoder);

      return accumulator;
    }

    @Override
    public Sketch<InputT> mergeAccumulators(Iterable<Sketch<InputT>> accumulators) {
      Iterator<Sketch<InputT>> it = accumulators.iterator();
      Sketch<InputT> first = it.next();
      CountMinSketch mergedSketches = first.sketch();
      try {
        while (it.hasNext()) {
          mergedSketches = CountMinSketch.merge(mergedSketches, it.next().sketch());
        }
      } catch (FrequencyMergeException e) {
        // Should never happen because every instantiated accumulator are of the same type.
        throw new IllegalStateException("The accumulators cannot be merged:" + e.getMessage());
      }

      return Sketch.create(mergedSketches);
    }

    /** Output the whole structure so it can be queried, reused or stored easily. */
    @Override
    public Sketch<InputT> extractOutput(Sketch<InputT> accumulator) {
      return accumulator;
    }

    @Override
    public Coder<Sketch<InputT>> getAccumulatorCoder(CoderRegistry registry, Coder inputCoder) {
      return new CountMinSketchCoder<>();
    }

    @Override
    public void populateDisplayData(DisplayData.Builder builder) {

View on GitHub (pinned to 12126d8942)