apache/beam · error · IOException

Failed to serialize and deserialize property '%s' with value

Error message

Failed to serialize and deserialize property '%s' with value '%s'

What it means

As a safety check when outputting PipelineOptions to JSON (e.g., for pipeline templates), Beam round-trips each property through serialize -> deserialize. If any property fails with Jackson, an IOException is thrown: "Failed to serialize and deserialize property '%s' with value '%s'", chaining the underlying exception.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java:885

      Map<String, Method> propertyToReadMethod = Maps.newHashMap();
      for (PropertyDescriptor descriptor : cache.getPropertyDescriptors(interfaces)) {
        if (descriptor.getReadMethod() != null) {
          propertyToReadMethod.put(descriptor.getName(), descriptor.getReadMethod());
        }
      }

      // Attempt to serialize and deserialize each property.
      for (Map.Entry<String, BoundValue> entry : options.entrySet()) {
        try {
          Object boundValue = entry.getValue().getValue();
          if (boundValue != null) {
            TokenBuffer buffer = new TokenBuffer(PipelineOptionsFactory.MAPPER, false);
            serializeEntry(entry.getKey(), boundValue, buffer, propertyToSerializer);
            Method method = propertyToReadMethod.get(entry.getKey());
            getValueFromJson(buffer.asParser().<JsonNode>readValueAsTree(), method);
          }
        } catch (Exception e) {
          throw new IOException(
              String.format(
                  "Failed to serialize and deserialize property '%s' with value '%s'",
                  entry.getKey(), entry.getValue().getValue()),
              e);
        }
      }
    }
  }

  static class Deserializer extends JsonDeserializer<PipelineOptions> {
    @Override
    public PipelineOptions deserialize(JsonParser jp, DeserializationContext ctxt)
        throws IOException {
      ObjectNode objectNode = jp.readValueAsTree();
      JsonNode rawOptionsNode = objectNode.get("options");

      Map<String, JsonNode> fields = Maps.newHashMap();
      if (rawOptionsNode != null && !rawOptionsNode.isNull()) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the chained cause to identify the failing property and Jackson error.
2. Change the option getter to a JSON-friendly type or pre-extract the needed fields into simple values.
3. Add Jackson annotations (@JsonTypeInfo/@JsonSubTypes) or a custom serializer/deserializer for the POJO.
4. Use ValueProvider-wrapped options or move complex config out of PipelineOptions entirely.
5. Verify with the same Beam version used at runtime that the round trip passes.

Example fix

// before
interface MyOptions extends PipelineOptions {
  MyComplexConfig getConfig(); // no Jackson support
}
// after
interface MyOptions extends PipelineOptions {
  @Default.String("{\"key\":\"value\"}")
  ValueProvider<String> getConfigJson(); // serialize config to JSON string
}

// Before creating a template, ensure option values are Jackson-serializable
MyOptions opts = PipelineOptionsFactory.fromArgs(args).as(MyOptions.class);
for (Method m : MyOptions.class.getMethods()) {
  if (m.getName().startsWith("get")) {
    Object v = m.invoke(opts); // must be primitives, Strings, collections,
                                // or POJOs with Jackson annotations
  }
}

try {
  pipeline.run(); // or options output
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to serialize and deserialize property")) {
    LOG.error("Option not JSON round-trippable: " + e.getMessage(), e.getCause());
  }
}

When it happens

Trigger: Calling PipelineOptions output/serialization (jsonFactory-based output / template creation) when a bound option value cannot survive a Jackson TokenBuffer serialize-then-parse round trip — e.g., an object lacking a usable Jackson serializer/deserializer, or a type that loses fidelity across JSON.

Common situations: Custom PipelineOptions getters returning complex types (POJOs without Jackson annotations, non-serializable objects); @JacksonDeserializable/@Default.Object setups whose class changed between versions; option values set to objects JSON cannot represent (streams, clients, lambdas).

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Failed to convert PipelineOptions to Protocol
• PipelineOptions objects are not serializable and should not
• Failed to read PipelineOptions from Protocol
• cannot encode a null Integer
• cannot encode a null Long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5d02e3aa3125b448. Report an issue: GitHub.