apache/beam · error · RuntimeException

Could not override the transforms with URNs

Error message

Could not override the transforms with URNs 

What it means

During PipelineTranslation.toProto(), when transform URNs are marked for override and upgradeTransforms is enabled, Beam delegates to TransformUpgrader.upgradeTransformsViaTransformService() to replace legacy transforms via a remote transform service. If that service call (or any part of it) throws, the exception is wrapped in a RuntimeException: 'Could not override the transforms with URNs ...'. It means the pipeline graph upgrade step failed before validation.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PipelineTranslation.java:122

        RunnerApi.Pipeline.newBuilder()
            .setComponents(components.toComponents())
            .addAllRequirements(components.requirements())
            .addAllRootTransformIds(rootIds)
            .build();
    if (!useDeprecatedViewTransforms) {
      // TODO(JIRA-5649): Don't even emit these transforms in the generated protos.
      res = elideDeprecatedViews(res);
    }

    List<String> urnsToOverride =
        pipeline.getOptions().as(ExternalTranslationOptions.class).getTransformsToOverride();
    if (urnsToOverride.size() > 0 && upgradeTransforms) {
      try (TransformUpgrader upgrader = TransformUpgrader.of()) {
        res =
            upgrader.upgradeTransformsViaTransformService(
                res, urnsToOverride, pipeline.getOptions());
      } catch (Exception e) {
        throw new RuntimeException(
            "Could not override the transforms with URNs " + urnsToOverride, e);
      }
    }

    // Validate that translation didn't produce an invalid pipeline.
    PipelineValidator.validate(res);
    return res;
  }

  private static RunnerApi.Pipeline elideDeprecatedViews(RunnerApi.Pipeline pipeline) {
    // Record data on CreateView operations.
    Set<String> viewTransforms = new HashSet<>();
    Map<String, String> viewOutputsToInputs = new HashMap<>();
    pipeline
        .getComponents()
        .getTransformsMap()
        .forEach(
            (transformId, transform) -> {

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the wrapped cause for the underlying failure (connectivity, unsupported URN, service error).
2. Verify the transform service endpoint/configuration (options/runner transform service settings) is reachable and correct.
3. Re-run with upgradeTransforms=false (or an empty urnsToOverride) so the pipeline is translated without the remote upgrade step, if the runner supports the legacy transforms.
4. Update the legacy transforms in pipeline code (replace deprecated PTransforms) so they no longer need overriding.

Example fix

// before
res = upgrader.upgradeTransformsViaTransformService(res, urnsToOverride, pipeline.getOptions());
// after — verify service reachability before relying on upgrades, or skip upgrade
if (urnsToOverride.size() > 0 && upgradeTransforms && transformServiceReachable(pipeline.getOptions())) {
  res = upgrader.upgradeTransformsViaTransformService(res, urnsToOverride, pipeline.getOptions());
}

if (!urnsToOverride.isEmpty()) { verifyTransformServiceReachable(pipeline.getOptions()); }

try { res = PipelineTranslation.toProto(pipeline, components, true, urnsToOverride); } catch (RuntimeException e) { LOG.warn("Transform upgrade failed", e); res = PipelineTranslation.toProto(pipeline, components, false, Collections.emptyList()); }

When it happens

Trigger: Calling PipelineTranslation.toProto(pipeline, sdkComponents, true, urnsToOverride) with a non-empty URN override list where TransformUpgrader.upgradeTransformsViaTransformService fails — transform service unreachable, rejects the transforms, times out, or returns an invalid result.

Common situations: Pipelines containing legacy transforms (e.g. old Read, Combine) that require upgrading, environments where the transform service endpoint is misconfigured or firewalled, network outages during job submission, or upgradeable URNs not supported by the configured transform service.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Either option TransformServiceAddress or option TransformSer
• Illegal access to pipeline after visitor traversal was compl
• Pipeline update will not be possible because the following t
• Unrecognized value for stable unique names:
• One or more ErrorHandlers aren't closed, and this pipeline c

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1e0dd9e751b9d1b5. Report an issue: GitHub.