apache/beam · error

apache beam installation failed in virtualenv

Error message

apache beam installation failed in virtualenv

What it means

Raised in expansionx.SetUpPythonEnvironment as a final sanity check: running `import apache_beam` with the venv's python failed after installation. It means apache_beam is installed but not importable, so the expansion service would not start correctly.

Solutions

1. Run `<venv>/bin/python -c "import apache_beam"` manually to see the ImportError/traceback
2. Fix the specific native dependency it reports (e.g. system libs) or switch to a supported OS/Python combo
3. Recreate the venv from scratch after deleting it to clear a corrupted install
4. Pin compatible transitive deps (e.g. protobuf) matching the installed apache_beam version

Example fix

// before
// ImportError: libcffi ... / protobuf conflict
// after
rm -rf venv && python3.10 -m venv venv && ./venv/bin/pip install apache_beam==2.60.0
./venv/bin/python -c "import apache_beam"

if err := exec.Command(venvPython, "-c", "import apache_beam").Run(); err != nil {
    return fmt.Errorf("apache_beam not importable in venv: %w", err)
}

python, err := expansionx.SetUpPythonEnvironment(ctx, py, pkg, extras)
if err != nil && strings.Contains(err.Error(), "import apache_beam") || strings.Contains(err.Error(), "installation failed in virtualenv") {
    return fmt.Errorf("run the import manually to see the traceback, fix native deps, and recreate the venv: %w", err)
}

When it happens

Trigger: startPythonExpansionService -> SetUpPythonEnvironment when the post-install verification `venvPython -c "import apache_beam"` exits non-zero — broken wheels, missing native libraries, or incompatible protobuf/dependency versions in the venv.

Common situations: Import-time errors like missing libso for grpc/protobuf, or 'google.protobuf' descriptor conflicts from mismatched protobuf versions; installing on an unsupported platform where a source build silently produced a broken package.

Related errors

• DaskRunner is not available. Please install…
• Failed to import hdfs. You can ensure it is installed by…
• Interactive runner not available, please install…
• Module not found in sys.modules
• A BigQuery table or a query must be specified

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/45a266e9e7b84b86. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download.go:487

		err = exec.Command(venvPython, "-m", "pip", "install", "--upgrade", "setuptools").Run()
		if err != nil {
			return "", errors.Wrap(err, "error upgrading setuptools")
		}
		err = exec.Command(venvPython, "-m", "pip", "install", beamPackage, "pyparsing==2.4.2").Run()
		if err != nil {
			return "", errors.Wrap(err, fmt.Sprintf("error installing beam package: %v", beamPackage))
		}
		if len(extraPackages) > 0 {
			cmd := []string{"-m", "pip", "install"}
			cmd = append(cmd, extraPackages...)
			err = exec.Command(venvPython, cmd...).Run()
			if err != nil {
				return "", errors.Wrap(err, "error installing dependencies")
			}
		}
		err = exec.Command(venvPython, "-c", "import apache_beam").Run()
		if err != nil {
			return "", errors.Wrap(err, "apache beam installation failed in virtualenv")
		}
	}
	return venvPython, nil
}

View on GitHub (pinned to 12126d8942)