apache/beam · error · IllegalStateException

Failed to encode key

Error message

Failed to encode key %s for side input id %s.

What it means

MultimapSideInput.encodeKey serializes a side-input map key with the key Coder before querying the runner for the key set. If the coder's encode() throws IOException (or the output stream fails), it wraps the failure in an IllegalStateException reporting the key and the side input id. This means the key is not encodable with the declared coder.

Solutions

1. Verify the key Coder matches the actual key type and that keys are non-null and encodable.
2. Check for null keys before reading the side input and skip or default them.
3. Re-derive the side input so its coder registry matches the read site (same PCollectionView/tag).
4. Fix or replace custom Coders that throw on valid runtime values.

Example fix

// before
V v = sideInput.get(maybeNullKey);
// after
V v = maybeNullKey == null ? defaultValue : sideInput.get(maybeNullKey);

if (k == null || !keyCoder.getType().isInstance(k)) {
  throw new IllegalArgumentException("Key not encodable for side input: " + k);
}

boolean encodableKey(K k, Coder<K> coder) {
  return k != null && coder.getType().isInstance(k);
}

try {
  V v = multimapSideInput.get(key);
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to encode key")) {
    LOG.error("Bad side input key {}: {}", key, e.getMessage());
  } else { throw e; }
}

When it happens

Trigger: Reading a multimap side input via get(k) (or encodedKey) where k cannot be encoded by the key Coder — e.g. coder mismatch, null key, or a coder whose encode assumes non-null/valid values.

Common situations: Side input built with a different key coder than the one used at read time after pipeline edits; null keys; custom coders throwing on certain values; serialization of mutated/invalid objects.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• buildDescriptor: failed to handle coder on stage
• Could not decode the default value with the provided coder
• err
• failed encoding key for
• Failed to encode values for multimap user state id

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/08831200371cc5cc. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java:180

                    .setTransformId(
                        keysRequest.getStateKey().getMultimapKeysSideInput().getTransformId())
                    .setSideInputId(
                        keysRequest.getStateKey().getMultimapKeysSideInput().getSideInputId())
                    .setWindow(keysRequest.getStateKey().getMultimapKeysSideInput().getWindow())
                    .setKey(encodedKey))
            .build();

    StateRequest request = keysRequest.toBuilder().setStateKey(stateKey).build();
    return StateFetchingIterators.readAllAndDecodeStartingFrom(
        Caches.subCache(cache, "ValuesForKey", encodedKey), beamFnStateClient, request, valueCoder);
  }

  private ByteString encodeKey(K k) {
    ByteStringOutputStream output = new ByteStringOutputStream();
    try {
      keyCoder.encode(k, output);
    } catch (IOException e) {
      throw new IllegalStateException(
          String.format(
              "Failed to encode key %s for side input id %s.",
              k, keysRequest.getStateKey().getMultimapKeysSideInput().getSideInputId()),
          e);
    }
    return output.toByteString();
  }
}

View on GitHub (pinned to 12126d8942)