apache/beam · error · java.lang.RuntimeException

Failed to convert PipelineOptions to Protocol

Error message

Failed to convert PipelineOptions to Protocol

What it means

After converting each pipeline option to a TreeNode keyed by URN, toProto merges the Jackson JSON into a protobuf Struct via JsonFormat.parser().merge(...). If any step throws IOException (typically Jackson serialization failure of the options map), it is wrapped in a RuntimeException with this message.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PipelineOptionsTranslation.java:82

            "Unable to convert pipeline options, please check for outdated jackson-core version in the classpath.");
      }

      Map<String, TreeNode> optionsUsingUrns = new HashMap<>();
      while (optionsEntries.hasNext()) {
        Map.Entry<String, JsonNode> entry = optionsEntries.next();
        optionsUsingUrns.put(
            PIPELINE_OPTIONS_URN_PREFIX
                + CaseFormat.LOWER_CAMEL.to(CaseFormat.LOWER_UNDERSCORE, entry.getKey())
                + PIPELINE_OPTIONS_URN_SUFFIX,
            entry.getValue());
      }

      // The JSON format of a Protobuf Struct is the JSON object that is equivalent to that struct
      // (with values encoded in a standard json-codeable manner). See Beam PR 3719 for more.
      JsonFormat.parser().merge(MAPPER.writeValueAsString(optionsUsingUrns), builder);
      return builder.build();
    } catch (IOException e) {
      throw new RuntimeException("Failed to convert PipelineOptions to Protocol", e);
    }
  }

  /** Converts the provided {@link Struct} into {@link PipelineOptions}. */
  public static PipelineOptions fromProto(Struct protoOptions) {
    try {
      Map<String, TreeNode> mapWithoutUrns = new HashMap<>();
      TreeNode rootOptions = MAPPER.readTree(JsonFormat.printer().print(protoOptions));
      Iterator<String> optionsKeys = rootOptions.fieldNames();
      while (optionsKeys.hasNext()) {
        String optionKey = optionsKeys.next();
        TreeNode optionValue = rootOptions.get(optionKey);
        mapWithoutUrns.put(
            CaseFormat.LOWER_UNDERSCORE.to(
                CaseFormat.LOWER_CAMEL,
                optionKey.substring(
                    PIPELINE_OPTIONS_URN_PREFIX.length(),
                    optionKey.length() - PIPELINE_OPTIONS_URN_SUFFIX.length())),

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the wrapped IOException cause to find which option value failed to serialize
2. Make the offending PipelineOptions property return a Jackson-friendly type (String, primitives, POJOs)
3. Annotate non-serializable option properties with @JsonIgnore and expose a serializable representation

Example fix

// before
MyConfig getConfig(); // unserializable third-party type
// after
@JsonIgnore MyConfig getConfig();
@JsonProperty String getConfigJson() { return serialize(getConfig()); }

try { Struct s = PipelineOptionsTranslation.toProto(options); } catch (RuntimeException e) { throw new IllegalStateException("PipelineOptions serialization failed: " + e.getCause(), e); }

When it happens

Trigger: Calling PipelineOptionsTranslation.toProto/toJson when MAPPER.writeValueAsString(optionsUsingUrns) throws IOException — e.g. an option value whose Jackson serialization fails (unserializable object, broken custom serializer).

Common situations: Custom PipelineOptions with getters returning objects Jackson cannot serialize; broken JsonSerialize annotations on option types; Jackson version mismatches at runtime.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Failed to serialize and deserialize property '%s' with value
• PipelineOptions objects are not serializable and should not
• Failed to read PipelineOptions from Protocol
• cannot encode a null Integer
• cannot encode a null Long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f456b3bd90308919. Report an issue: GitHub.