apache/beam · error · RuntimeException

Expected the ManagedTransform schema to include a field…

Error message

Expected the ManagedTransform schema to include a field named 'transform_identifier' but received ${configRow}

What it means

getTransformUniqueID expects the Managed transform's config Row schema to include a field named 'transform_identifier' which uniquely identifies the underlying transform; if absent it throws a RuntimeException including the received configRow.

Solutions

1. Include a 'transform_identifier' field in the ManagedTransform config schema and set its value.
2. Align client Beam SDK version with the expansion service's expected Managed schema.
3. Print configRow.getSchema().getFieldNames() (included in the message) to see actual fields.
4. Regenerate the config via the current Managed Transform API instead of hand-built Rows.

Example fix

// before
Schema configSchema = Schema.of(Field.of("provider", Schema.FieldType.STRING));
// after
Schema configSchema = Schema.of(
    Field.of("provider", Schema.FieldType.STRING),
    Field.of("transform_identifier", Schema.FieldType.STRING));

Row configRow = decode(payload);
if (!configRow.getSchema().getFieldNames().contains("transform_identifier"))
  throw new IllegalArgumentException("Config missing transform_identifier");

Optional<Object> transformId(Row row) { return row.getSchema().getFieldNames().contains("transform_identifier") ? Optional.ofNullable(row.getValue("transform_identifier")) : Optional.empty(); }

try { String id = provider.getTransformUniqueID(payload); } catch (RuntimeException e) { log.error("Managed config invalid: {}", e.getMessage()); throw new ManagedConfigException(e); }

When it happens

Trigger: Submitting a managed-transform request whose config Row schema omits 'transform_identifier' or names it differently; config built by an incompatible Managed schema version.

Common situations: Beam version skew between client and expansion service; hand-built config Rows; YAML configs missing the identifier key.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• ArrowFlightIO.write(): e.getMessage()
• BeamRowMapper does not have support for fields of type
• can't generate row coder for type
• Cannot find Spanner table.
• cannot make schema for type

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0cb0122a9e2ca1d8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/TransformProvider.java:151

        .equals(spec.getUrn())) {
      ExternalTransforms.SchemaTransformPayload payload;
      try {
        payload = ExternalTransforms.SchemaTransformPayload.parseFrom(spec.getPayload());
        if (PTransformTranslation.MANAGED_TRANSFORM_URN.equals(payload.getIdentifier())) {
          try {
            // ManagedSchemaTransform includes a schema field transform_identifier that includes the
            // underlying schema
            // transform ID so we special case that here.
            Row configRow =
                RowCoder.of(SchemaTranslation.schemaFromProto(payload.getConfigurationSchema()))
                    .decode(new ByteArrayInputStream(payload.getConfigurationRow().toByteArray()));

            for (String field : configRow.getSchema().getFieldNames()) {
              if (field.equals("transform_identifier")) {
                return configRow.getValue(field);
              }
            }
            throw new RuntimeException(
                "Expected the ManagedTransform schema to include a field named "
                    + "'transform_identifier' but received "
                    + configRow);
          } catch (IOException e) {
            throw new RuntimeException(e);
          }
        } else {
          return payload.getIdentifier();
        }
      } catch (InvalidProtocolBufferException e) {
        throw new IllegalArgumentException(
            "Invalid payload type for URN "
                + BeamUrns.getUrn(ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM),
            e);
      }
    }
    return spec.getUrn();
  }

View on GitHub (pinned to 12126d8942)