apache/beam · error · java.lang.RuntimeException

Failed to build transform from spec

Error message

Failed to build transform from spec %s: %s

What it means

For schema-based (external configuration) transforms, ExpansionService.getTransform() parses the spec payload into a config object via transformBuilder.buildExternal(payloadToConfig(...)); any failure is rethrown as this RuntimeException with the spec and original message. It indicates the external config could not be parsed or the builder rejected it.

Solutions

1. Inspect the 'Caused by' to see whether parsing or buildExternal failed
2. Compare the payload schema against the Java configuration class's registered schema
3. Update the calling side to send exactly the fields the config class declares
4. Rebuild/redeploy the expansion service if the config class changed recently

Example fix

// before: external payload with unexpected field type
{'retries': 'three'}
// after
{'retries': 3}

// validate payload fields against the Java config class schema before expansion
Schema configSchema = SchemaRegistry.createDefault().getSchema(ConfigClass.class);
if (!configSchema.getFieldNames().equals(sentFieldNames)) throw new IllegalArgumentException("payload schema mismatch");

try { transform = service.getTransform(spec, options); } catch (RuntimeException e) { if (e.getMessage().startsWith("Failed to build transform from spec")) { /* reconcile payload schema with config class */ } throw e; }

When it happens

Trigger: Calling getTransform on a FunctionSpec handled by a schema-based transform builder where ExternalConfigurationPayload parsing or payloadToConfig conversion throws (wrong schema, missing fields, reflective construction failure).

Common situations: Cross-language pipeline sending config values that do not match the Java config class schema; extra/missing parameters in the external payload; type mismatches (string vs int) after schema evolution.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Schema in expansion request payload is not assignable to…
• A transform cannot be initiated using the provided config…
• Cannot find Spanner table.
• Cannot merge schemas with different numbers of fields…
• Cannot merge two types: +fieldType1.getTypeName()+ and…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/daef33e9d01788f0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java:337

  }

  private static class TransformProviderForBuilder implements TransformProvider {

    private final ExternalTransformBuilder transformBuilder;

    private TransformProviderForBuilder(ExternalTransformBuilder transformBuilder) {
      this.transformBuilder = transformBuilder;
    }

    @Override
    public PTransform getTransform(RunnerApi.FunctionSpec spec, PipelineOptions options) {
      try {
        Class configClass = getConfigClass(transformBuilder);
        return transformBuilder.buildExternal(
            payloadToConfig(
                ExternalConfigurationPayload.parseFrom(spec.getPayload()), configClass));
      } catch (Exception e) {
        throw new RuntimeException(
            String.format("Failed to build transform from spec %s: %s", spec, e.getMessage()), e);
      }
    }

    @Override
    public List<String> getDependencies(RunnerApi.FunctionSpec spec, PipelineOptions options) {
      try {
        Class configClass = getConfigClass(transformBuilder);
        Optional<List<String>> dependencies =
            transformBuilder.getDependencies(
                payloadToConfig(
                    ExternalConfigurationPayload.parseFrom(spec.getPayload()), configClass),
                options);
        return dependencies.orElseGet(() -> TransformProvider.super.getDependencies(spec, options));
      } catch (Exception e) {
        throw new RuntimeException(
            String.format("Failed to get dependencies for spec %s", spec), e);
      }

View on GitHub (pinned to 12126d8942)