{"record":{"id":"26f4b3ebc8c9b666","repo":"apache/beam","slug":"the-input-value-cannot-be-encoded-e-getmessage-26f4b3","errorCode":null,"errorMessage":"The input value cannot be encoded: ${e.getMessage()}","messagePattern":"The input value cannot be encoded: (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java","lineNumber":487,"sourceCode":"\n    abstract int width();\n\n    abstract CountMinSketch sketch();\n\n    public void add(T element, long count, Coder<T> coder) {\n      sketch().add(hashElement(element, coder), count);\n    }\n\n    public void add(T element, Coder<T> coder) {\n      add(element, 1L, coder);\n    }\n\n    private long hashElement(T element, Coder<T> coder) {\n      try {\n        byte[] elemBytes = CoderUtils.encodeToByteArray(coder, element);\n        return Hashing.murmur3_128().hashBytes(elemBytes).asLong();\n      } catch (CoderException e) {\n        throw new IllegalStateException(\"The input value cannot be encoded: \" + e.getMessage(), e);\n      }\n    }\n\n    /**\n     * Utility class to retrieve the estimate frequency of an element from a {@link CountMinSketch}.\n     */\n    public long estimateCount(T element, Coder<T> coder) {\n      return sketch().estimateCount(hashElement(element, coder));\n    }\n  }\n\n  /** Coder for {@link CountMinSketch} class. */\n  static class CountMinSketchCoder<T> extends CustomCoder<Sketch<T>> {\n\n    private static final ByteArrayCoder BYTE_ARRAY_CODER = ByteArrayCoder.of();\n\n    @Override\n    public void encode(Sketch<T> value, OutputStream outStream) throws IOException {","sourceCodeStart":469,"sourceCodeEnd":505,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java#L469-L505","documentation":"In SketchFrequencies, hashElement encodes each incoming element to bytes with CoderUtils.encodeToByteArray before hashing it into the Count-Min Sketch. If the element's Coder throws a CoderException (the element doesn't conform to the coder's expectations), the code rethrows it as an IllegalStateException with this message. This means the value cannot be serialized for hashing, so it cannot be inserted into or looked up in the sketch.","triggerScenarios":"Calling add(value) or estimateCount(value) on a SketchFrequencies transform when the value cannot be encoded by the registered Coder<T> — e.g. a coder whose encode() rejects the value, a coder mismatched with the actual runtime type, or a custom coder throwing CoderException for that particular element.","commonSituations":"Using a coder that doesn't match the element's runtime class after a pipeline refactor; a custom Coder with strict validation rejecting nulls or out-of-range fields; relying on inferred coders that became incompatible after a schema/type change.","solutions":["Verify the Coder<T> supplied to SketchFrequencies matches the actual runtime type of the elements being added.","Log the offending element (e.getMessage() contains the CoderException detail) and fix or clean the element so it is encodable.","Write or update a custom Coder for the element type and test it with CoderTester/coder round-trip tests.","Catch the IllegalStateException upstream in a DoFn and route bad elements to a dead-letter output instead of failing the bundle."],"exampleFix":"// before\nPCollection<KV<String, Long>> est = sketched.apply(SketchFrequencies.estimateCount(badElements));\n\n// after: ensure the coder matches the element type\nPCollection<MyType> typed = badElements.setCoder(MyTypeCoder.of());\nPCollection<KV<MyType, Long>> est =\n    typed.apply(SketchFrequencies.<MyType>builder().build())\n         .apply(SketchFrequencies.estimateCount());","handlingStrategy":"try-catch","validationCode":"// Before adding, verify the element encodes with its coder\ntry {\n  CoderUtils.encodeToByteArray(coder, element);\n} catch (CoderException e) {\n  throw new IllegalStateException(\"Element not encodable by \" + coder + \": \" + element, e);\n}","typeGuard":null,"tryCatchPattern":"try {\n  sketch = elements.apply(SketchFrequencies.<T>builder().setCoder(coder).build());\n} catch (IllegalStateException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"The input value cannot be encoded\")) {\n    LOG.error(\"Element failed coder encoding: {}\", e.getCause(), e);\n    // route to dead-letter or fix coder\n  } else {\n    throw e;\n  }\n}","preventionTips":["Always pair SketchFrequencies with a coder that matches the element's runtime type.","Round-trip test custom coders (encode/decode equality) before deploying.","Include a representative sample of real elements in coder unit tests."],"tags":["java","beam","encoding","coder","sketching"],"backgroundTag":"json-marshal-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}