apache/beam · error · IOException

Unable to find CoderTranslator for known Coder

Error message

Unable to find CoderTranslator for known Coder

What it means

CoderTranslation.toKnownCoder throws this IOException when a Coder class has no registered CoderTranslator despite being treated as a known coder — i.e. the translation registry lookup returned null while serializing the coder to its RunnerApi proto form.

Solutions

1. Ensure META-INF/services/org.apache.beam.sdk.util.construction.CoderTranslatorRegistrar exists in the jar and lists all registrars
2. Rebuild the fat jar preserving service files (don't merge/shade them away; check shade plugin transformation filters)
3. Register your custom coder with a CoderTranslatorRegistrar implementation
4. Verify with CoderTranslation.verifyModelCodersRegistered() early in job startup

Example fix

// before
public class MyCoders { } // no registrar
// after
public class MyCoderRegistrar implements CoderTranslatorRegistrar {
  public Map<Class<? extends Coder<?>>, String> getCoderURNs() { ... }
  public List<? extends CoderTranslator<?>> getCoderTranslators() { ... }
}
// plus META-INF/services entry naming MyCoderRegistrar

CoderTranslation.verifyModelCodersRegistered(); // fail fast at startup
// and for custom coders ensure a registrar covers coder.getClass()

try {
  return CoderTranslation.toProto(coder, components);
} catch (IOException e) {
  if (e.getMessage().contains("Unable to find CoderTranslator")) {
    throw new IllegalStateException("Coder " + coder.getClass() + " lacks a registered translator", e);
  }
  throw e;
}

When it happens

Trigger: Serializing a pipeline (SdkComponents/register) with a custom Coder whose class maps into the known-coder path but lacks a CoderTranslator registered via the CoderTranslatorRegistrar ServiceLoader.

Common situations: Fat/uber jars built without the META-INF/services CoderTranslatorRegistrar entries; custom coder subclasses overriding the standard ones; shading that strips service files.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

• cannot encode a null Count-min Sketch
• cannot encode a null HyperLogLogPlus sketch
• cannot encode a null Integer
• cannot encode a null PredictionResult
• cannot encode a null SolrDocument

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/55e761d37b48cf21. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/CoderTranslation.java:185

    }

    return toCustomCoder(coder);
  }

  private static RunnerApi.Coder toUnknownCoderWrapper(UnknownCoderWrapper coder) {
    return RunnerApi.Coder.newBuilder()
        .setSpec(
            FunctionSpec.newBuilder()
                .setUrn(coder.getUrn())
                .setPayload(ByteString.copyFrom(coder.getPayload())))
        .build();
  }

  private static RunnerApi.Coder toKnownCoder(Coder<?> coder, SdkComponents components)
      throws IOException {
    CoderTranslator translator = getCoderTranslator(coder.getClass());
    if (translator == null) {
      throw new IOException("Unable to find CoderTranslator for known Coder");
    }
    List<String> componentIds = registerComponents(coder, translator, components);
    return RunnerApi.Coder.newBuilder()
        .addAllComponentCoderIds(componentIds)
        .setSpec(
            FunctionSpec.newBuilder()
                .setUrn(getKnownCoderUrns().get(coder.getClass()))
                .setPayload(ByteString.copyFrom(translator.getPayload(coder))))
        .build();
  }

  private static <T extends Coder<?>> List<String> registerComponents(
      T coder, CoderTranslator<T> translator, SdkComponents components) throws IOException {
    List<String> componentIds = new ArrayList<>();
    for (Coder<?> component : translator.getComponents(coder)) {
      componentIds.add(components.registerCoder(component));
    }
    return componentIds;

View on GitHub (pinned to 12126d8942)