apache/beam · error · RuntimeError

Could not generate external transform wrappers due to error

Error message

Could not generate external transform wrappers due to error: {}

What it means

After locating gen_xlang_wrappers.py, setup.py runs it via subprocess with check=True; a nonzero exit (CalledProcessError) is re-raised as RuntimeError embedding the wrapper generator's stderr. The generator itself failed (e.g. missing deps, bad config, template errors).

Solutions

Inspect err.stderr in the message for the generator's actual error
Run `python gen_xlang_wrappers.py --output_file <path>` manually to see full traceback
Install generator dependencies (e.g. pip install pyyaml jinja2)
Regenerate against a matching standard_external_transforms.yaml for your Beam version

Example fix

# before
$ python setup.py build
RuntimeError: Could not generate external transform wrappers due to error: b'No module named yaml'
# after
$ pip install pyyaml
$ python sdks/python/gen_xlang_wrappers.py  # verify success
$ python setup.py build

Defensive patterns

Strategy: try-catch

Validate before calling

import subprocess
r = subprocess.run(['python', 'gen_xlang_wrappers.py', '--help'], capture_output=True)
if r.returncode != 0:
    fix_generator_env(r.stderr)  # missing deps etc.

Try / catch

try:
    build()
except RuntimeError as e:
    if 'Could not generate external transform wrappers' in str(e):
        install_generator_deps(); run_gen_xlang_wrappers_manually(); build()
    else:
        raise

Prevention

Install the generator's dependencies (pyyaml, jinja2) in build environments
Run the generator standalone once to validate the environment
Keep standard_external_transforms.yaml in sync with your Beam version

When it happens

Trigger: Building apache_beam from source when the gen_xlang_wrappers.py subprocess exits nonzero — missing jinja2/YAML deps, invalid standard_external_transforms.yaml, or errors in the generated wrapper templates.

Common situations: Fresh environments without the generator's dependencies installed; Beam version upgrades changing the YAML schema; CI sandboxes blocking resources the generator needs.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/849893acd9cdc15d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/setup.py:321

        message += ' could not be found'
        raise RuntimeError(message)
      else:
        logging.info(
            'Skipping external transform wrapper generation as they '
            'are already generated.')
      return
    subprocess.run([
        sys.executable,
        os.path.join(sdk_dir, 'gen_xlang_wrappers.py'),
        '--cleanup',
        '--transforms-config-source',
        os.path.join(
            os.path.dirname(sdk_dir), 'standard_external_transforms.yaml')
    ],
                   capture_output=True,
                   check=True)
  except subprocess.CalledProcessError as err:
    raise RuntimeError(
        'Could not generate external transform wrappers due to '
        'error: {}'.format(err.stderr))


def get_portability_package_data():
  files = []
  portability_dir = Path(__file__).parent / 'apache_beam' / \
                    'portability' / 'api'
  for ext in ['*.pyi', '*.yaml']:
    files.extend(
        str(p.relative_to(portability_dir.parent.parent))
        for p in portability_dir.rglob(ext))

  return files


python_requires = '>=3.10'

View on GitHub (pinned to 12126d8942)