apache/beam · error · ImportError

Failed to import hdfs. You can ensure it is installed by ins

Error message

Failed to import hdfs. You can ensure it is installed by installing the hadoop beam extra

What it means

HadoopFileSystem.__init__ requires the third-party 'hdfs' Python package. If the module failed to import (it is None here), the class raises ImportError telling the user to install the 'hadoop' Beam extra, since the HDFS filesystem cannot function without this client library.

Source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:115

    self._handle.__exit__(None, None, None)
    self._handle = None
    self._handle_context = None


class HadoopFileSystem(FileSystem):
  """``FileSystem`` implementation that supports HDFS.

  URL arguments to methods expect strings starting with ``hdfs://``.
  """
  def __init__(self, pipeline_options):
    """Initializes a connection to HDFS.

    Connection configuration is done by passing pipeline options.
    See :class:`~apache_beam.options.pipeline_options.HadoopFileSystemOptions`.
    """
    super().__init__(pipeline_options)
    if hdfs is None:
      raise ImportError(
          'Failed to import hdfs. You can ensure it is '
          'installed by installing the hadoop beam extra')
    logging.getLogger('hdfs.client').setLevel(logging.WARN)
    if pipeline_options is None:
      raise ValueError('pipeline_options is not set')
    if isinstance(pipeline_options, PipelineOptions):
      hdfs_options = pipeline_options.view_as(HadoopFileSystemOptions)
      hdfs_host = hdfs_options.hdfs_host
      hdfs_port = hdfs_options.hdfs_port
      hdfs_user = hdfs_options.hdfs_user
      self._full_urls = hdfs_options.hdfs_full_urls
    else:
      hdfs_host = pipeline_options.get('hdfs_host')
      hdfs_port = pipeline_options.get('hdfs_port')
      hdfs_user = pipeline_options.get('hdfs_user')
      self._full_urls = pipeline_options.get('hdfs_full_urls', False)

    if hdfs_host is None:

View on GitHub (pinned to 12126d8942)

Solutions

1. Install the extra: pip install 'apache_beam[hadoop]'.
2. Or install the client directly: pip install hdfs.
3. Verify with `python -c "import hdfs"` before running the pipeline.
4. Pin/reinstall apache_beam with extras if packaging stripped dependencies.

Example fix

// before
pip install apache_beam
// after
pip install "apache_beam[hadoop]"

try:
    import hdfs
    hdfs_available = True
except ImportError:
    hdfs_available = False

None

try:
    fs = HadoopFileSystem(pipeline_options=opts)
except ImportError as e:
    if 'Failed to import hdfs' in str(e):
        raise SystemExit("Install with: pip install 'apache_beam[hadoop]'")
    raise

When it happens

Trigger: Using HDFS sources/sinks (e.g. 'hdfs://...' paths or HadoopFileSystem) in an environment where apache_beam was installed without the hadoop extra, so `import hdfs` failed and hdfs is None.

Common situations: Minimal docker images or Airflow workers with only the base apache_beam package; CI environments missing extras;Beam installed from source without `[hadoop]`.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• DaskRunner is not available. Please install apache_beam[dask
• Module {parts[0]} not found in sys.modules
• google-cloud-bigquery-storage is required for ReadBigQueryCh
• pyarrow is required for ReadBigQueryChangeHistory. Install i
• pipeline_options is not set

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5d1de82489ad4d00. Report an issue: GitHub.