apache/beam · error · IllegalStateException

The input value cannot be encoded: ${e.getMessage()}

Error message

The input value cannot be encoded: ${e.getMessage()}

What it means

ApproximateDistinctFn.addInput encodes each incoming record to bytes with the configured inputCoder before offering them to the HyperLogLogPlus sketch. If the record cannot be encoded (CoderException), this IllegalStateException is thrown, since the combiner cannot proceed without a byte representation.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java:459

      checkArgument(
          (sp > this.p && sp < 32) || (sp == 0),
          "Expected: p <= sp <= 32." + "Actual: p = %s, sp = %s",
          this.p,
          sp);
      return new ApproximateDistinctFn<>(this.p, sp, this.inputCoder);
    }

    @Override
    public HyperLogLogPlus createAccumulator() {
      return new HyperLogLogPlus(p, sp);
    }

    @Override
    public HyperLogLogPlus addInput(HyperLogLogPlus acc, InputT record) {
      try {
        acc.offer(CoderUtils.encodeToByteArray(inputCoder, record));
      } catch (CoderException e) {
        throw new IllegalStateException("The input value cannot be encoded: " + e.getMessage(), e);
      }
      return acc;
    }

    /** Output the whole structure so it can be queried, reused or stored easily. */
    @Override
    public HyperLogLogPlus extractOutput(HyperLogLogPlus accumulator) {
      return accumulator;
    }

    @Override
    public HyperLogLogPlus mergeAccumulators(Iterable<HyperLogLogPlus> accumulators) {
      HyperLogLogPlus mergedAccum = createAccumulator();
      for (HyperLogLogPlus accum : accumulators) {
        try {
          mergedAccum.addAll(accum);
        } catch (CardinalityMergeException e) {
          // Should never happen because only HyperLogLogPlus accumulators are instantiated.

View on GitHub (pinned to 12126d8942)

Solutions

1. Filter out null/incompatible elements with Filter.by(...) before the ApproximateDistinct transform.
2. Ensure the coder passed to create matches the PCollection's actual element type (Coder of the PCollection).
3. Fix the custom coder's encode method for the failing value; inspect the CoderException cause for details.

Example fix

// before
PCollection<String> items = ...; // may contain nulls
items.apply(ApproximateDistinct.<String>globally().create(StringUtf8Coder.of()));
// after
items.apply(Filter.by(v -> v != null))
     .apply(ApproximateDistinct.<String>globally().create(StringUtf8Coder.of()));

// Filter unencodable elements upstream
PCollection<String> clean = input.apply(Filter.by(v -> v != null));

try { acc.offer(CoderUtils.encodeToByteArray(inputCoder, record)); } catch (CoderException e) { LOG.error("unencodable element: {}", record, e); throw e; }

When it happens

Trigger: Streaming an element into the sketch whose runtime type does not match the Coder passed to create (e.g. coder is for String but a null or non-String element arrives); a custom coder failing on this particular value (e.g. unsupported nested type).

Common situations: Null elements reaching the combiner with a coder that cannot encode null; type changes upstream after a refactor so elements no longer match the declared coder; a custom coder throwing on edge-case values.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• The input value cannot be encoded: ${e.getMessage()}
• cannot encode a null HyperLogLogPlus sketch
• Unable to encode element '" + value + "' with coder '" + thi
• Invalid encoded string length: {}
• cannot encode a null String

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3ab2863e526394b9. Report an issue: GitHub.