apache/beam · error · ValueError

hdfs_port is not set

Error message

hdfs_port is not set

What it means

HadoopFileSystem validates that hdfs_port is provided when building its hdfs.InsecureClient. A missing hdfs_port raises ValueError 'hdfs_port is not set', since the client needs an HTTP port to reach the HDFS WebHDFS endpoint.

Source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:136

    logging.getLogger('hdfs.client').setLevel(logging.WARN)
    if pipeline_options is None:
      raise ValueError('pipeline_options is not set')
    if isinstance(pipeline_options, PipelineOptions):
      hdfs_options = pipeline_options.view_as(HadoopFileSystemOptions)
      hdfs_host = hdfs_options.hdfs_host
      hdfs_port = hdfs_options.hdfs_port
      hdfs_user = hdfs_options.hdfs_user
      self._full_urls = hdfs_options.hdfs_full_urls
    else:
      hdfs_host = pipeline_options.get('hdfs_host')
      hdfs_port = pipeline_options.get('hdfs_port')
      hdfs_user = pipeline_options.get('hdfs_user')
      self._full_urls = pipeline_options.get('hdfs_full_urls', False)

    if hdfs_host is None:
      raise ValueError('hdfs_host is not set')
    if hdfs_port is None:
      raise ValueError('hdfs_port is not set')
    if hdfs_user is None:
      raise ValueError('hdfs_user is not set')
    if not isinstance(self._full_urls, bool):
      raise ValueError(
          'hdfs_full_urls should be bool, got: %s', self._full_urls)
    self._hdfs_client = hdfs.InsecureClient(
        'http://%s:%s' % (hdfs_host, str(hdfs_port)), user=hdfs_user)

  @classmethod
  def scheme(cls):
    return 'hdfs'

  def _parse_url(self, url):
    """Verifies that url begins with hdfs:// prefix, strips it and adds a
    leading /.

    Parsing behavior is determined by HadoopFileSystemOptions.hdfs_full_urls.

View on GitHub (pinned to 12126d8942)

Solutions

1. Add --hdfs_port=<port> (WebHDFS port, e.g. 50070 or 9870 for Hadoop 3.x) to the pipeline options.
2. Include 'hdfs_port': 50070 in the dict of options.
3. Confirm the correct WebHDFS port against your cluster's hdfs-site.xml (dfs.namenode.http-address).
4. Validate all of hdfs_host/hdfs_port/hdfs_user together before launching.

Example fix

// before
PipelineOptions(['--hdfs_host=namenode.example.com', '--hdfs_user=hduser'])
// after
PipelineOptions(['--hdfs_host=namenode.example.com', '--hdfs_port=50070', '--hdfs_user=hduser'])

def require_hdfs_port(opts):
    v = (opts.get('hdfs_port') if isinstance(opts, dict)
         else opts.view_as(HadoopFileSystemOptions).hdfs_port)
    if not v:
        raise ValueError('Set --hdfs_port (WebHDFS port, e.g. 50070) before launching')
    return v

None

try:
    fs = HadoopFileSystem(pipeline_options=opts)
except ValueError as e:
    if 'hdfs_port is not set' in str(e):
        raise SystemExit('Add --hdfs_port=<webhdfs-port> to pipeline options')
    raise

When it happens

Trigger: Initializing HadoopFileSystem with options that include hdfs_host and hdfs_user but no 'hdfs_port' / --hdfs_port flag.

Common situations: Config copied from examples that omit the port; WebHDFS port (typically 50070/9870) unknown to the user; dict-based options missing the key.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• pipeline_options is not set
• hdfs_host is not set
• hdfs_user is not set
• Cannot create a temporary directory for root path prefix %s.
• MatchContinuously(timestamp_cursor=True) deduplicates, so it

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e76c4c5a94de4188. Report an issue: GitHub.