apache/beam · error · RuntimeException

Failed to convert PipelineOptions to JSON

Error message

Failed to convert PipelineOptions to JSON

What it means

PipelineOptionsTranslation.toJson() serializes a PipelineOptions object to a JSON string by first converting it to a RunnerApi.ProtocolBuffer message and then printing it with JsonFormat.printer(). If the protobuf-to-JSON printing step throws InvalidProtocolBufferException, the method wraps it in a RuntimeException with this message. This indicates the generated proto could not be rendered as JSON, which is normally an internal/invariant failure rather than a user input problem.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PipelineOptionsTranslation.java:135

        // Legacy options.
        return MAPPER.readValue(optionsJson, PipelineOptions.class);
      } else {
        // Fn Options with namespace and version.
        Struct.Builder builder = Struct.newBuilder();
        JsonFormat.parser().merge(optionsJson, builder);
        return fromProto(builder.build());
      }
    } catch (IOException e) {
      throw new RuntimeException("Failed to read PipelineOptions from JSON", e);
    }
  }

  /** Converts the provided {@link PipelineOptions} into Json{@link String}. */
  public static String toJson(PipelineOptions options) {
    try {
      return JsonFormat.printer().print(toProto(options));
    } catch (InvalidProtocolBufferException e) {
      throw new RuntimeException("Failed to convert PipelineOptions to JSON", e);
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the wrapped cause (e.getCause(), the InvalidProtocolBufferException) to identify which field/proto fails JSON conversion.
2. Simplify or fix the custom PipelineOptions / payload translator producing the offending proto payload.
3. Verify SDK and runner versions match; upgrade Beam to the latest patch release for the version in use.
4. As a workaround, serialize options via PipelineOptionsTranslation.toProto(options) directly or rely on the runner's own serialization path.

Example fix

// before
String json = PipelineOptionsTranslation.toJson(options);
// after
try {
  String json = PipelineOptionsTranslation.toJson(options);
} catch (RuntimeException e) {
  LOG.error("Options proto -> JSON failed; proto dump for debug:", e.getCause());
  String protoDebug = TextFormat.printer().printToString(PipelineOptionsTranslation.toProto(options));
}

if (options == null) throw new IllegalArgumentException("PipelineOptions required");

try { String json = PipelineOptionsTranslation.toJson(options); } catch (RuntimeException e) { handleSerializationFailure(e.getCause(), options); }

When it happens

Trigger: Calling PipelineOptionsTranslation.toJson(options) where JsonFormat.printer().print(toProto(options)) throws InvalidProtocolBufferException — e.g. the converted FunctionSpec/proto contains data the JSON printer cannot serialize (malformed Any payloads or fields incompatible with the canonical JSON mapping).

Common situations: Building runner pipelines programmatically, submitting jobs to a runner (Flink/Dataflow/Spark) that serializes options, using custom PipelineOptions registrations or custom payload translators that produce unusual proto contents, or Beam version mismatches between SDK and runner producing protos the printer rejects.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Cannot provide SerializableCoder because {} does not impleme
• Java Serialization may be non-deterministic.
• cannot encode a null String
• cannot encode a null Integer
• cannot encode a null ValueKind

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5c7130b6c336cac9. Report an issue: GitHub.