apache/beam · error · IllegalArgumentException

Invalid payload type for URN ${BeamUrns.getUrn(ExternalTrans

Error message

Invalid payload type for URN ${BeamUrns.getUrn(ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM)}

What it means

Thrown by getTransformUniqueID when the SchemaTransform payload bytes cannot be parsed as an ExternalTransforms.SchemaTransformPayload protobuf. The expansion service requires SCHEMA_TRANSFORM urns to carry a valid SchemaTransformPayload; a malformed or wrong-type payload makes unique-ID derivation impossible.

Source

Thrown at sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/TransformProvider.java:162

                    .decode(new ByteArrayInputStream(payload.getConfigurationRow().toByteArray()));

            for (String field : configRow.getSchema().getFieldNames()) {
              if (field.equals("transform_identifier")) {
                return configRow.getValue(field);
              }
            }
            throw new RuntimeException(
                "Expected the ManagedTransform schema to include a field named "
                    + "'transform_identifier' but received "
                    + configRow);
          } catch (IOException e) {
            throw new RuntimeException(e);
          }
        } else {
          return payload.getIdentifier();
        }
      } catch (InvalidProtocolBufferException e) {
        throw new IllegalArgumentException(
            "Invalid payload type for URN "
                + BeamUrns.getUrn(ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM),
            e);
      }
    }
    return spec.getUrn();
  }

  default List<String> getDependencies(RunnerApi.FunctionSpec spec, PipelineOptions options) {
    ExpansionServiceConfig config =
        options.as(ExpansionServiceOptions.class).getExpansionServiceConfig();
    String transformUniqueID = getTransformUniqueID(spec);

    boolean isManagedExpansion = false;
    if (getUrn(ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM).equals(spec.getUrn())) {
      try {
        ExternalTransforms.SchemaTransformPayload schemaTransformPayload =
            ExternalTransforms.SchemaTransformPayload.parseFrom(spec.getPayload());

View on GitHub (pinned to 12126d8942)

Solutions

1. Align the Beam SDK version of the submitting pipeline with the expansion service version so payload serialization matches
2. Verify the transform's ExternalTransformRegistrar produces a valid SchemaTransformPayload (test expansion locally)
3. Inspect the payload bytes; if hand-built, regenerate via SchemaTransformPayload.newBuilder() and build()
4. Catch IllegalArgumentException in the expansion caller and log the payload identifier for diagnosis

Example fix

// before
String id = expansionService.transformUniqueID(spec); // spec payload not a SchemaTransformPayload
// after
try {
  String id = expansionService.transformUniqueID(spec);
} catch (IllegalArgumentException e) {
  // fall back to spec.getUrn() or fail with a clear message about version mismatch
  String id = spec.getUrn();
}

boolean isValidSchemaTransformPayload(RunnerApi.FunctionSpec spec) {
  try {
    ExternalTransforms.SchemaTransformPayload.parseFrom(spec.getPayload());
    return true;
  } catch (InvalidProtocolBufferException e) { return false; }
}

try { id = provider.transformUniqueID(spec); } catch (IllegalArgumentException e) { log.error("bad SCHEMA_TRANSFORM payload", e); id = spec.getUrn(); }

When it happens

Trigger: Calling transformUniqueID/getTransformUniqueID with an ExpansionRequest whose spec has urn SCHEMA_TRANSFORM but whose payload was built by a different Beam version, hand-crafted, truncated, or is not a SchemaTransformPayload message.

Common situations: Cross-version Beam SDK/gradle-version mismatch between pipeline submitting the transform and the expansion service jar; corrupt payloads from custom ExternalTransformRegistrar implementations; manually constructed RunnerApi.FunctionSpec.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Unable to infer data schema from configuration proto.
• Timing number 0b" + timingNumber.toString(2) + " has more th
• No proto encoding for PaneInfoCoder, always part of Windowed
• Failed to validate %s
• Could not decode bytes as message

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d64a61ba8b190f8f. Report an issue: GitHub.