apache/beam · error · java.lang.RuntimeException

Failed to read PipelineOptions from Protocol

Error message

Failed to read PipelineOptions from Protocol

What it means

fromProto converts a protobuf Struct of pipeline options back into PipelineOptions by round-tripping through Jackson (writeValueAsString then readValue into PipelineOptions). If the Jackson read/write throws IOException, Beam wraps it in a RuntimeException with this message, indicating the proto options could not be materialized into a PipelineOptions instance.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PipelineOptionsTranslation.java:107

      Map<String, TreeNode> mapWithoutUrns = new HashMap<>();
      TreeNode rootOptions = MAPPER.readTree(JsonFormat.printer().print(protoOptions));
      Iterator<String> optionsKeys = rootOptions.fieldNames();
      while (optionsKeys.hasNext()) {
        String optionKey = optionsKeys.next();
        TreeNode optionValue = rootOptions.get(optionKey);
        mapWithoutUrns.put(
            CaseFormat.LOWER_UNDERSCORE.to(
                CaseFormat.LOWER_CAMEL,
                optionKey.substring(
                    PIPELINE_OPTIONS_URN_PREFIX.length(),
                    optionKey.length() - PIPELINE_OPTIONS_URN_SUFFIX.length())),
            optionValue);
      }
      return MAPPER.readValue(
          MAPPER.writeValueAsString(ImmutableMap.of("options", mapWithoutUrns)),
          PipelineOptions.class);
    } catch (IOException e) {
      throw new RuntimeException("Failed to read PipelineOptions from Protocol", e);
    }
  }

  /** Converts the provided Json{@link String} into {@link PipelineOptions}. */
  public static PipelineOptions fromJson(String optionsJson) {
    try {
      Map<String, Object> probingOptionsMap =
          MAPPER.readValue(optionsJson, new TypeReference<Map<String, Object>>() {});
      if (probingOptionsMap.containsKey("options")) {
        // Legacy options.
        return MAPPER.readValue(optionsJson, PipelineOptions.class);
      } else {
        // Fn Options with namespace and version.
        Struct.Builder builder = Struct.newBuilder();
        JsonFormat.parser().merge(optionsJson, builder);
        return fromProto(builder.build());
      }
    } catch (IOException e) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the wrapped IOException for the specific property deserialization failure
2. Ensure both pipeline ends use the same PipelineOptions classes registered via AutoService(PipelineOptionsRegistrar)
3. Verify the Struct contents match the expected {options: {urn: value}} layout produced by toProto

Example fix

// before
fromProto(structWithUnknownOptions) // java.io.IOException at readValue
// after
Struct fixed = alignOptionKeysWithRegisteredInterfaces(struct); // then fromProto(fixed)

try { PipelineOptions o = PipelineOptionsTranslation.fromProto(struct); } catch (RuntimeException e) { log.error("proto->options failed", e.getCause()); throw e; }

When it happens

Trigger: Calling PipelineOptionsTranslation.fromProto/fromJson with a Struct whose values don't match the registered PipelineOptions bean properties (unknown types, malformed nodes) causing MAPPER.readValue to fail.

Common situations: Options produced by a different Beam version or SDK with incompatible option names/types; hand-edited pipeline JSON; missing PipelineOptionsRegistrar registrations for custom option classes.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• Failed to serialize and deserialize property '%s' with value
• Failed to convert PipelineOptions to Protocol
• Property [%s] is marked with contradictory annotations. Foun
• All inherited interfaces of [%s] should inherit from the Pip
• Method [%s] has multiple definitions %s with different retur

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6fbe71fad95601fe. Report an issue: GitHub.