apache/beam · error · NotSerializableException

PipelineOptions objects are not serializable and should not

Error message

PipelineOptions objects are not serializable and should not be embedded into transforms (did you capture a PipelineOptions object in a field or in an anonymous class?). Instead, if you're using a DoFn, access PipelineOptions at runtime via ProcessContext/StartBundleContext/FinishBundleContext.getPipelineOptions(), or pre-extract necessary fields from PipelineOptions at pipeline construction time.

What it means

Apache Beam's PipelineOptions are dynamic JDK proxies backed by ProxyInvocationHandler, which is deliberately not serializable. When a PipelineOptions proxy is captured in a DoFn field, an anonymous class, or a transform closure, Java serialization during pipeline construction fails and this NotSerializableException is thrown with guidance on the correct patterns.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java:253

      BoundValue prev =
          options.put(
              properties.settersToPropertyNames.get(methodName),
              BoundValue.fromExplicitOption(args[0]));
      if (prev == null ? args[0] != null : !Objects.equals(args[0], prev.getValue())) {
        revision.incrementAndGet();
      }
      return Void.TYPE;
    }
    throw new RuntimeException(
        "Unknown method [" + method + "] invoked with args [" + Arrays.toString(args) + "].");
  }

  public String getOptionName(Method method) {
    return computedProperties.gettersToPropertyNames.get(method.getName());
  }

  private void writeObject(java.io.ObjectOutputStream stream) throws IOException {
    throw new NotSerializableException(
        "PipelineOptions objects are not serializable and should not be embedded into transforms "
            + "(did you capture a PipelineOptions object in a field or in an anonymous class?). "
            + "Instead, if you're using a DoFn, access PipelineOptions at runtime "
            + "via ProcessContext/StartBundleContext/FinishBundleContext.getPipelineOptions(), "
            + "or pre-extract necessary fields from PipelineOptions "
            + "at pipeline construction time.");
  }

  /** Track whether options values are explicitly set, or retrieved from defaults. */
  @AutoValue
  abstract static class BoundValue {

    abstract @Nullable Object getValue();

    abstract boolean isDefault();

    private static BoundValue of(@Nullable Object value, boolean isDefault) {
      return new AutoValue_ProxyInvocationHandler_BoundValue(value, isDefault);

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove the PipelineOptions field/capture from the DoFn or transform.
2. Access options at runtime via ProcessContext.getPipelineOptions() (or StartBundleContext/FinishBundleContext) inside the DoFn.
3. Pre-extract only the needed primitive/config fields from PipelineOptions at pipeline construction time and store those.
4. If a value is needed remotely, wrap it in a ValueProvider or StaticValueProvider of the extracted value.

Example fix

// before
class MyDoFn extends DoFn<String, String> {
  private final MyOptions options;
  MyDoFn(MyOptions options) { this.options = options; } // not serializable
}
// after
class MyDoFn extends DoFn<String, String> {
  @ProcessElement
  public void process(ProcessContext ctx) {
    MyOptions options = ctx.getPipelineOptions().as(MyOptions.class);
  }
}

// Before submitting, audit DoFn fields for captured PipelineOptions
for (Field f : myDoFn.getClass().getDeclaredFields()) {
  if (PipelineOptions.class.isAssignableFrom(f.getType())) {
    throw new IllegalStateException("Field captures PipelineOptions: " + f.getName());
  }
}

static boolean isPipelineOptionsCaptured(Class<?> fnClass) {
  for (Field f : fnClass.getDeclaredFields()) {
    if (PipelineOptions.class.isAssignableFrom(f.getType())) return true;
  }
  return false;
}

try {
  pipeline.run();
} catch (NotSerializableException e) {
  // message names ProxyInvocationHandler; refactor DoFn to use
  // ctx.getPipelineOptions() or pre-extracted fields
}

When it happens

Trigger: Java serialization (ObjectOutputStream.writeObject) reaches a ProxyInvocationHandler instance — typically because a PipelineOptions object was stored in a member field of a DoFn/transform, or captured by an anonymous inner class or lambda in the pipeline graph.

Common situations: Assigning options to a field in a DoFn constructor; capturing options inside an anonymous ParDo/MapElements; passing options into a transform's constructor and storing it; Beam runners serializing the DAG during submit.

Related errors

• Failed to serialize and deserialize property '%s' with value
• Failed to convert PipelineOptions to Protocol
• cannot encode a null Integer
• cannot encode a null Long
• cannot encode a null Short

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/acc9c4c395d3b450. Report an issue: GitHub.