{"record":{"id":"5d1de82489ad4d00","repo":"apache/beam","slug":"failed-to-import-hdfs-you-can-ensure-it-is-installed-by","errorCode":null,"errorMessage":"Failed to import hdfs. You can ensure it is installed by installing the hadoop beam extra","messagePattern":"Failed to import hdfs\\. You can ensure it is installed by installing the hadoop beam extra","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":115,"sourceCode":"    self._handle.__exit__(None, None, None)\n    self._handle = None\n    self._handle_context = None\n\n\nclass HadoopFileSystem(FileSystem):\n  \"\"\"``FileSystem`` implementation that supports HDFS.\n\n  URL arguments to methods expect strings starting with ``hdfs://``.\n  \"\"\"\n  def __init__(self, pipeline_options):\n    \"\"\"Initializes a connection to HDFS.\n\n    Connection configuration is done by passing pipeline options.\n    See :class:`~apache_beam.options.pipeline_options.HadoopFileSystemOptions`.\n    \"\"\"\n    super().__init__(pipeline_options)\n    if hdfs is None:\n      raise ImportError(\n          'Failed to import hdfs. You can ensure it is '\n          'installed by installing the hadoop beam extra')\n    logging.getLogger('hdfs.client').setLevel(logging.WARN)\n    if pipeline_options is None:\n      raise ValueError('pipeline_options is not set')\n    if isinstance(pipeline_options, PipelineOptions):\n      hdfs_options = pipeline_options.view_as(HadoopFileSystemOptions)\n      hdfs_host = hdfs_options.hdfs_host\n      hdfs_port = hdfs_options.hdfs_port\n      hdfs_user = hdfs_options.hdfs_user\n      self._full_urls = hdfs_options.hdfs_full_urls\n    else:\n      hdfs_host = pipeline_options.get('hdfs_host')\n      hdfs_port = pipeline_options.get('hdfs_port')\n      hdfs_user = pipeline_options.get('hdfs_user')\n      self._full_urls = pipeline_options.get('hdfs_full_urls', False)\n\n    if hdfs_host is None:","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L97-L133","documentation":"HadoopFileSystem.__init__ requires the third-party 'hdfs' Python package. If the module failed to import (it is None here), the class raises ImportError telling the user to install the 'hadoop' Beam extra, since the HDFS filesystem cannot function without this client library.","triggerScenarios":"Using HDFS sources/sinks (e.g. 'hdfs://...' paths or HadoopFileSystem) in an environment where apache_beam was installed without the hadoop extra, so `import hdfs` failed and hdfs is None.","commonSituations":"Minimal docker images or Airflow workers with only the base apache_beam package; CI environments missing extras;Beam installed from source without `[hadoop]`.","solutions":["Install the extra: pip install 'apache_beam[hadoop]'.","Or install the client directly: pip install hdfs.","Verify with `python -c \"import hdfs\"` before running the pipeline.","Pin/reinstall apache_beam with extras if packaging stripped dependencies."],"exampleFix":"// before\npip install apache_beam\n// after\npip install \"apache_beam[hadoop]\"","handlingStrategy":"fallback","validationCode":"try:\n    import hdfs\n    hdfs_available = True\nexcept ImportError:\n    hdfs_available = False","typeGuard":"None","tryCatchPattern":"try:\n    fs = HadoopFileSystem(pipeline_options=opts)\nexcept ImportError as e:\n    if 'Failed to import hdfs' in str(e):\n        raise SystemExit(\"Install with: pip install 'apache_beam[hadoop]'\")\n    raise","preventionTips":["Install apache_beam with the [hadoop] extra wherever the pipeline runs","Include the extra in Dockerfiles and requirements files","Smoke-test 'import hdfs' in CI before deploying","Keep base beam install and extras consistent across environments"],"tags":["hdfs","import-error","python","dependency","apache-beam"],"backgroundTag":"missing-optional-dependency","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}