apache/beam · error · ValueError

pipeline_options is not set

Error message

pipeline_options is not set

What it means

HadoopFileSystem.__init__ needs connection configuration supplied via pipeline options (hdfs_host, hdfs_port, hdfs_user). If pipeline_options is None it cannot read any configuration and raises ValueError 'pipeline_options is not set'.

Source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:120

class HadoopFileSystem(FileSystem):
  """``FileSystem`` implementation that supports HDFS.

  URL arguments to methods expect strings starting with ``hdfs://``.
  """
  def __init__(self, pipeline_options):
    """Initializes a connection to HDFS.

    Connection configuration is done by passing pipeline options.
    See :class:`~apache_beam.options.pipeline_options.HadoopFileSystemOptions`.
    """
    super().__init__(pipeline_options)
    if hdfs is None:
      raise ImportError(
          'Failed to import hdfs. You can ensure it is '
          'installed by installing the hadoop beam extra')
    logging.getLogger('hdfs.client').setLevel(logging.WARN)
    if pipeline_options is None:
      raise ValueError('pipeline_options is not set')
    if isinstance(pipeline_options, PipelineOptions):
      hdfs_options = pipeline_options.view_as(HadoopFileSystemOptions)
      hdfs_host = hdfs_options.hdfs_host
      hdfs_port = hdfs_options.hdfs_port
      hdfs_user = hdfs_options.hdfs_user
      self._full_urls = hdfs_options.hdfs_full_urls
    else:
      hdfs_host = pipeline_options.get('hdfs_host')
      hdfs_port = pipeline_options.get('hdfs_port')
      hdfs_user = pipeline_options.get('hdfs_user')
      self._full_urls = pipeline_options.get('hdfs_full_urls', False)

    if hdfs_host is None:
      raise ValueError('hdfs_host is not set')
    if hdfs_port is None:
      raise ValueError('hdfs_port is not set')
    if hdfs_user is None:
      raise ValueError('hdfs_user is not set')

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass a PipelineOptions (or dict) containing hdfs_host, hdfs_port, hdfs_user.
2. Provide CLI flags --hdfs_host, --hdfs_port, --hdfs_user when launching the pipeline.
3. Wrap raw dict values in PipelineOptions as the class accepts both forms.
4. Add an assertion/default pipeline_options before constructing the filesystem.

Example fix

// before
fs = HadoopFileSystem(pipeline_options=None)
// after
opts = PipelineOptions(['--hdfs_host=namenode', '--hdfs_port=8020', '--hdfs_user=hduser'])
fs = HadoopFileSystem(pipeline_options=opts)

def has_hdfs_options(opts) -> bool:
    if opts is None:
        return False
    if isinstance(opts, dict):
        return 'hdfs_host' in opts
    from apache_beam.options.pipeline_options import PipelineOptions, HadoopFileSystemOptions
    return opts.view_as(HadoopFileSystemOptions).hdfs_host is not None

None

try:
    fs = HadoopFileSystem(pipeline_options=opts)
except ValueError as e:
    if str(e) == 'pipeline_options is not set':
        fs = HadoopFileSystem(pipeline_options=build_default_hdfs_options())
    else:
        raise

When it happens

Trigger: Constructing HadoopFileSystem(pipeline_options=None) directly, or a filesystem registration path that passes no options object.

Common situations: Programmatic use of the HDFS filesystem without setting HadoopFileSystemOptions; tests instantiating the class with defaults; pipeline launched without --hdfs_host etc. so options resolution yields None.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• hdfs_host is not set
• hdfs_port is not set
• hdfs_user is not set
• Cannot create a temporary directory for root path prefix %s.
• MatchContinuously(timestamp_cursor=True) deduplicates, so it

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/addf0921ede157d6. Report an issue: GitHub.