apache/beam · error · RuntimeError

protobuf files are not generated. Please generate pb2 files

Error message

protobuf files are not generated. Please generate pb2 files

What it means

setup.py's generate_protos_first can skip running gen_protos.py when pb2 files already exist, but if the skip flag is set and no *_pb2.py files are found under apache_beam/portability/api, it raises RuntimeError. It refuses to package a SDK whose protobuf bindings were never generated.

Solutions

1. Run `python setup.py generate_protos_first` without the skip flag, or run `python gen_protos.py` first
2. Verify apache_beam/portability/api contains *_pb2.py files: ls apache_beam/portability/api/*_pb2.py
3. Install protoc and grpcio-tools prerequisites if gen_protos.py failed silently earlier
4. Use the official PyPI sdist which ships pre-generated pb2 files

Example fix

# before
$ python setup.py py_sdist --skip_proto_generation
RuntimeError: protobuf files are not generated. Please generate pb2 files
# after
$ python gen_protos.py
$ python setup.py py_sdist --skip_proto_generation

import glob, os
pb2 = glob.glob(os.path.join('apache_beam', 'portability', 'api', '*_pb2.py'))
if not pb2:
    run_gen_protos_first()  # python gen_protos.py

try:
    build_package()
except RuntimeError as e:
    if 'protobuf files are not generated' in str(e):
        subprocess.run(['python', 'gen_protos.py'], check=True)
        build_package()
    else:
        raise

When it happens

Trigger: Building/installing apache_beam with the skip-proto-generation option while apache_beam/portability/api contains no *_pb2.py files — e.g. a clean checkout, a wiped source tree, or an sdist missing generated files.

Common situations: Cloning the repo and building without first running gen_protos.py; pip installing from a git checkout; CI cleaning generated artifacts between steps.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

• Skipping proto generation as they are already generated.
• Could not generate external transform wrappers due to error
• Could not generate protos due to error
• Cannot convert from nanoseconds to microseconds because…
• cannot convert to micro seconds

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/19d155120ad2e119. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/setup.py:247

# We must generate protos after setup_requires are installed.
def generate_protos_first():
  try:
    # Pyproject toml build happens in isolated environemnts. In those envs,
    # gen_protos is unable to get imported. so we run a subprocess call.
    cwd = os.path.abspath(os.path.dirname(__file__))
    # when pip install <>.tar.gz gets called, if gen_protos.py is not available
    # in the sdist,then the proto files would have already been generated. So we
    # skip proto generation in that case.
    if not os.path.exists(os.path.join(cwd, 'gen_protos.py')):
      # make sure we already generated protos
      pb2_files = list(
          find_by_ext(
              os.path.join(cwd, 'apache_beam', 'portability', 'api'),
              '_pb2.py'))
      if not pb2_files:
        raise RuntimeError(
            'protobuf files are not generated. '
            'Please generate pb2 files')

      warnings.warn('Skipping proto generation as they are already generated.')
      return
    out = subprocess.run(
        [sys.executable, os.path.join(cwd, 'gen_protos.py'), '--no-force'],
        capture_output=True,
        check=True)
    print(out.stdout)
  except subprocess.CalledProcessError as err:
    raise RuntimeError('Could not generate protos due to error: %s', err.stderr)


def copy_tests_from_docs():
  python_root = os.path.abspath(os.path.dirname(__file__))
  docs_src = os.path.normpath(
      os.path.join(

View on GitHub (pinned to 12126d8942)