apache/beam · error · RuntimeException

Unable to perform expansion for transform

Error message

Unable to perform expansion for transform 

What it means

Thrown by SplittableParDoExpander.getReplacement when expanding a SPLITTABLE_PROCESS_URN transform raises an IOException while constructing the replacement composite (building restriction coders, PairWithRestriction, ProcessSizedElementsAndRestrictions transforms). It marks that the automatic expansion of the splittable DoFn transform could not be performed.

Solutions

1. Inspect the chained IOException cause — it usually points to a serialization failure of the DoFn, coder, or restriction type.
2. Ensure the element/restriction coders are registered and serializable (check Coder verification with getCoderArguments).
3. Verify all classes referenced by the DoFn exist on the classpath used by the expansion code.
4. If your runner cannot support splittable DoFn, use a non-splittable source or register an expansion fallback.

Example fix

// before: unserializable custom coder
new MyCoder() // holds non-serializable field
// after
class MyCoder extends CustomCoder<MyT> {
  void encode(MyT v, OutputStream out) { ... } // stateless, serializable
}

// ensure DoFn and coders are serializable before expansion
try { java.io.ObjectOutputStream oos = new java.io.ObjectOutputStream(new java.io.ByteArrayOutputStream()); oos.writeObject(dofn); oos.close(); } catch (java.io.IOException e) { throw new IllegalArgumentException("DoFn is not serializable; expansion will fail", e); }

boolean isSerializable(Object o) { return o instanceof java.io.Serializable || org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(o) != null; }

try { replacement = expander.getReplacement(transformNode); } catch (RuntimeException e) { log.error("Splittable ParDo expansion failed for {}: {}", transformId, e.getCause() != null ? e.getCause().getMessage() : e.getMessage(), e); throw e; }

When it happens

Trigger: Calling getReplacement for a splittable ParDo transform where building sub-transform protos throws IOException — most commonly serializing the DoFn or coder payload via SerializableUtils/proto serialization inside the expansion.

Common situations: Splittable DoFns (e.g. file-based sources) with non-serializable restriction coders or DoFn state; classpath issues where referenced classes cannot be resolved during payload serialization; custom runners lacking splittable-DoFn expansion support.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• cannot encode a null Count-min Sketch
• cannot encode a null Integer
• cannot encode a null String
• cannot encode a null T-Digest sketch
• Cannot encode a null value.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/063b600edc370f88. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/graph/SplittableParDoExpander.java:312

              generateUniquePCollectonName(
                  splittableParDo.getUniqueName() + "/ProcessSizedElementsAndRestrictions",
                  existingComponents));
          processSizedElementsAndRestrictions.setSpec(
              FunctionSpec.newBuilder()
                  .setUrn(
                      PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN)
                  .setPayload(splittableParDo.getSpec().getPayload()));
          processSizedElementsAndRestrictions.setEnvironmentId(splittableParDo.getEnvironmentId());
          rval.getComponentsBuilder()
              .putTransforms(
                  processSizedElementsAndRestrictionsId,
                  processSizedElementsAndRestrictions.build());
        }
        newCompositeRoot.addSubtransforms(processSizedElementsAndRestrictionsId);
        rval.setPtransform(newCompositeRoot);
        return rval.build();
      } catch (IOException e) {
        throw new RuntimeException("Unable to perform expansion for transform " + transformId, e);
      }
    }
  }

  private static String getOrAddDoubleCoder(
      ComponentsOrBuilder existingComponents, MessageWithComponents.Builder out) {
    for (Map.Entry<String, Coder> coder : existingComponents.getCodersMap().entrySet()) {
      if (ModelCoders.DOUBLE_CODER_URN.equals(coder.getValue().getSpec().getUrn())) {
        return coder.getKey();
      }
    }
    String doubleCoderId = generateUniqueId("DoubleCoder", existingComponents::containsCoders);
    out.getComponentsBuilder()
        .putCoders(
            doubleCoderId,
            Coder.newBuilder()
                .setSpec(FunctionSpec.newBuilder().setUrn(ModelCoders.DOUBLE_CODER_URN))
                .build());

View on GitHub (pinned to 12126d8942)