apache/beam · error · ValueError

Incompatible environments

Error message

Incompatible environments: '%s' != '%s'

What it means

During stage fusion, translations._merge_environments checks whether two adjacent stages use identical execution environments. If both stages have environments and they differ (SDK version, container image, dependencies), fusion is refused with this ValueError because the merged stage could not run in a single worker environment.

Solutions

1. Unify environments: use the same container_image/SDK version for the transforms you want fused
2. Apply custom environments consistently to the whole pipeline rather than to individual transforms
3. If differing environments are intentional, accept non-fused stages — remove or adjust the fusion optimization, or insert an explicit boundary
4. Compare the two environment protos in the error message (newline-flattened) to see exactly which field differs

Example fix

// before: custom env on one transform only
PCollection<int> out = transform.setInput(p).setEnvironment(customEnv).expand();
// after: same env pipeline-wide (or none, letting the default apply)
PCollection<int> out = transform.setInput(p).expand();

envs = {t.spec.environment_id for t in pipeline.components.transforms.values() if t.spec.environment_id}
if len(envs) > 1:
    print('Differing environments:', [pipeline.components.protos.environments[e] for e in envs])

try:
    optimized = pipeline.run()
except ValueError as e:
    if 'Incompatible environments' in str(e):
        log.error('Stage fusion refused: %s', e)  # envs differ; unify container_image/deps
    raise

When it happens

Trigger: Building an optimized pipeline where a consumer transform declares a different environment than its producer — e.g. different docker container_image, different SDK harness version, or one transform with custom setup.py dependencies.

Common situations: Mixing transforms from different SDK versions in one pipeline, custom container images applied to only part of a pipeline, or external pipeline fragments (e.g. cross-language transforms) with their own environment merged into a pipeline.

Related errors

• Could not find Python executable.
• Java is not correctly installed in JAVA_HOME=
• Java must be installed on this system to use this…
• No proto encoding for PaneInfoCoder, always part of…
• no python installation found. If you use a custom container…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/335e04fe80ac08df. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py:183

        must_follow,
        downstream_side_inputs)

  @staticmethod
  def _extract_environment(transform):
    # type: (beam_runner_api_pb2.PTransform) -> Optional[str]
    environment = transform.environment_id
    return environment if environment else None

  @staticmethod
  def _merge_environments(env1, env2):
    # type: (Optional[str], Optional[str]) -> Optional[str]
    if env1 is None:
      return env2
    elif env2 is None:
      return env1
    else:
      if env1 != env2:
        raise ValueError(
            "Incompatible environments: '%s' != '%s'" %
            (str(env1).replace('\n', ' '), str(env2).replace('\n', ' ')))
      return env1

  def can_fuse(self, consumer, context):
    # type: (Stage, TransformContext) -> bool
    try:
      self._merge_environments(self.environment, consumer.environment)
    except ValueError:
      return False

    def no_overlap(a, b):
      return not a or not b or not a.intersection(b)

    return (
        not consumer.forced_root and not self in consumer.must_follow and
        self.is_all_sdk_urns(context) and consumer.is_all_sdk_urns(context) and
        no_overlap(self.downstream_side_inputs, consumer.side_inputs()))

View on GitHub (pinned to 12126d8942)