apache/beam · error · IllegalArgumentException

Could not find a transform with the ID

Error message

Could not find a transform with the ID 

What it means

updateTransformViaTransformService looks up the transform to upgrade by ID in the runner API pipeline proto. If the ID is absent from the pipeline's transforms map, the upgrade cannot proceed and it throws IllegalArgumentException.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/TransformUpgrader.java:197

    }

    if (service != null) {
      service.shutdown();
    }

    return pipeline;
  }

  private RunnerApi.Pipeline updateTransformViaTransformService(
      RunnerApi.Pipeline runnerAPIpipeline,
      String transformId,
      Endpoints.ApiServiceDescriptor transformServiceEndpoint,
      PipelineOptions options)
      throws IOException {
    RunnerApi.PTransform transformToUpgrade =
        runnerAPIpipeline.getComponents().getTransformsMap().get(transformId);
    if (transformToUpgrade == null) {
      throw new IllegalArgumentException("Could not find a transform with the ID " + transformId);
    }

    byte[] payloadBytes = null;

    if (!transformToUpgrade.getSpec().getUrn().equals(BeamUrns.getUrn(SCHEMA_TRANSFORM))) {
      ByteString configRowBytes =
          transformToUpgrade.getAnnotationsOrThrow(
              BeamUrns.getConstant(ExternalTransforms.Annotations.Enum.CONFIG_ROW_KEY));
      ByteString configRowSchemaBytes =
          transformToUpgrade.getAnnotationsOrThrow(
              BeamUrns.getConstant(ExternalTransforms.Annotations.Enum.CONFIG_ROW_SCHEMA_KEY));
      SchemaApi.Schema configRowSchemaProto =
          SchemaApi.Schema.parseFrom(configRowSchemaBytes.toByteArray());
      payloadBytes =
          ExternalTransforms.ExternalConfigurationPayload.newBuilder()
              .setSchema(configRowSchemaProto)
              .setPayload(configRowBytes)
              .build()

View on GitHub (pinned to 12126d8942)

Solutions

1. Use the exact composite/transform ID present in the pipeline proto (inspect via pipeline.toString() or RunnerApi dump)
2. Re-run upgrader before other rewrites that might rename transforms
3. Remove the stale ID from transformsToOverride

Example fix

// before
options.setTransformsToOverride(ImmutableMap.of("old_id", "urn:beam:...")); // id not in pipeline
// after
// find real id from pipeline proto, e.g. "External/Read" 
options.setTransformsToOverride(ImmutableMap.of("External/Read", "urn:beam:..."));

// Check IDs against the pipeline proto before upgrading:
Map<String, RunnerApi.PTransform> transforms =
    RunnerApi.Pipeline.parseFrom(pipelineBytes).getComponents().getTransformsMap();
for (String id : requestedIds) {
  if (!transforms.containsKey(id)) throw new IllegalStateException("No transform with ID: " + id);
}

try { pipeline.run(); } catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Could not find a transform with the ID")) { /* fix the ID */ }
  else throw e;
}

When it happens

Trigger: Passing a transformId in transformsToOverride map that doesn't exist in the current pipeline proto (stale ID, renamed transform, or wrong pipeline).

Common situations: Transform IDs changed after earlier rewriting/optimization passes; overriding transforms in a different pipeline than the one the IDs came from; typos in transform IDs.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Failed to validate transform %s
• A transform cannot be initiated using the provided config ro
• Failed to build transform %s from spec %s: %s
• Illegal access to pipeline after visitor traversal was compl
• Pipeline update will not be possible because the following t

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6c38bf5d432f0e15. Report an issue: GitHub.