apache/beam · error · IllegalStateException

The input value cannot be encoded: ${e.getMessage()}

Error message

The input value cannot be encoded: ${e.getMessage()}

What it means

In SketchFrequencies, hashElement encodes each incoming element to bytes with CoderUtils.encodeToByteArray before hashing it into the Count-Min Sketch. If the element's Coder throws a CoderException (the element doesn't conform to the coder's expectations), the code rethrows it as an IllegalStateException with this message. This means the value cannot be serialized for hashing, so it cannot be inserted into or looked up in the sketch.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java:487

    abstract int width();

    abstract CountMinSketch sketch();

    public void add(T element, long count, Coder<T> coder) {
      sketch().add(hashElement(element, coder), count);
    }

    public void add(T element, Coder<T> coder) {
      add(element, 1L, coder);
    }

    private long hashElement(T element, Coder<T> coder) {
      try {
        byte[] elemBytes = CoderUtils.encodeToByteArray(coder, element);
        return Hashing.murmur3_128().hashBytes(elemBytes).asLong();
      } catch (CoderException e) {
        throw new IllegalStateException("The input value cannot be encoded: " + e.getMessage(), e);
      }
    }

    /**
     * Utility class to retrieve the estimate frequency of an element from a {@link CountMinSketch}.
     */
    public long estimateCount(T element, Coder<T> coder) {
      return sketch().estimateCount(hashElement(element, coder));
    }
  }

  /** Coder for {@link CountMinSketch} class. */
  static class CountMinSketchCoder<T> extends CustomCoder<Sketch<T>> {

    private static final ByteArrayCoder BYTE_ARRAY_CODER = ByteArrayCoder.of();

    @Override
    public void encode(Sketch<T> value, OutputStream outStream) throws IOException {

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the Coder<T> supplied to SketchFrequencies matches the actual runtime type of the elements being added.
2. Log the offending element (e.getMessage() contains the CoderException detail) and fix or clean the element so it is encodable.
3. Write or update a custom Coder for the element type and test it with CoderTester/coder round-trip tests.
4. Catch the IllegalStateException upstream in a DoFn and route bad elements to a dead-letter output instead of failing the bundle.

Example fix

// before
PCollection<KV<String, Long>> est = sketched.apply(SketchFrequencies.estimateCount(badElements));

// after: ensure the coder matches the element type
PCollection<MyType> typed = badElements.setCoder(MyTypeCoder.of());
PCollection<KV<MyType, Long>> est =
    typed.apply(SketchFrequencies.<MyType>builder().build())
         .apply(SketchFrequencies.estimateCount());

// Before adding, verify the element encodes with its coder
try {
  CoderUtils.encodeToByteArray(coder, element);
} catch (CoderException e) {
  throw new IllegalStateException("Element not encodable by " + coder + ": " + element, e);
}

try {
  sketch = elements.apply(SketchFrequencies.<T>builder().setCoder(coder).build());
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("The input value cannot be encoded")) {
    LOG.error("Element failed coder encoding: {}", e.getCause(), e);
    // route to dead-letter or fix coder
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling add(value) or estimateCount(value) on a SketchFrequencies transform when the value cannot be encoded by the registered Coder<T> — e.g. a coder whose encode() rejects the value, a coder mismatched with the actual runtime type, or a custom coder throwing CoderException for that particular element.

Common situations: Using a coder that doesn't match the element's runtime class after a pipeline refactor; a custom Coder with strict validation rejecting nulls or out-of-range fields; relying on inferred coders that became incompatible after a schema/type change.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• The input value cannot be encoded: ${e.getMessage()}
• cannot encode a null HyperLogLogPlus sketch
• Unable to encode element '" + value + "' with coder '" + thi
• Invalid encoded string length: {}
• cannot encode a null String

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/26f4b3ebc8c9b666. Report an issue: GitHub.