apache/beam · error · ValueError

hdfs_full_urls should be bool, got

Error message

hdfs_full_urls should be bool, got: %s

What it means

hdfs_full_urls controls whether paths are treated as complete WebHDFS URLs including the host. HadoopFileSystem enforces that it is a bool; any other type (string 'true', int, None from an explicit get default mismatch) triggers ValueError 'hdfs_full_urls should be bool, got: %s'.

Solutions

  1. Cast the value to bool before constructing: bool(my_full_urls) only if it truly represents a boolean.
  2. Pass True or False literals, not 'true'/'false' strings.
  3. Use the typed HadoopFileSystemOptions option so Beam parses the flag properly.
  4. Coerce in a wrapper: opts['hdfs_full_urls'] = str(opts['hdfs_full_urls']).lower() == 'true'.

Example fix

// before
options = {'hdfs_full_urls': 'true'}
// after
options = {'hdfs_full_urls': True}
Defensive patterns

Strategy: type-guard

Validate before calling

def as_bool(v):
    if isinstance(v, bool):
        return v
    if isinstance(v, str):
        return v.lower() == 'true'
    raise ValueError('hdfs_full_urls must be bool')
opts['hdfs_full_urls'] = as_bool(opts.get('hdfs_full_urls', False))

Type guard

def is_real_bool(x: object) -> bool:
    return isinstance(x, bool)

Try / catch

try:
    fs = HadoopFileSystem(pipeline_options=opts)
except ValueError as e:
    if 'hdfs_full_urls should be bool' in str(e):
        opts['hdfs_full_urls'] = str(opts['hdfs_full_urls']).lower() == 'true'
        fs = HadoopFileSystem(pipeline_options=opts)
    else:
        raise

Prevention

When it happens

Trigger: Passing --hdfs_full_urls as a non-boolean value (e.g. 'yes', '1' in a dict it stays a str), or a dict options entry with 'hdfs_full_urls': 'true' / 1 instead of True/False.

Common situations: YAML/JSON configs that supply the flag as a string; users passing 0/1; templated pipelines that interpolate the flag as text.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2ba2933a37fb887d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:140

      hdfs_options = pipeline_options.view_as(HadoopFileSystemOptions)
      hdfs_host = hdfs_options.hdfs_host
      hdfs_port = hdfs_options.hdfs_port
      hdfs_user = hdfs_options.hdfs_user
      self._full_urls = hdfs_options.hdfs_full_urls
    else:
      hdfs_host = pipeline_options.get('hdfs_host')
      hdfs_port = pipeline_options.get('hdfs_port')
      hdfs_user = pipeline_options.get('hdfs_user')
      self._full_urls = pipeline_options.get('hdfs_full_urls', False)

    if hdfs_host is None:
      raise ValueError('hdfs_host is not set')
    if hdfs_port is None:
      raise ValueError('hdfs_port is not set')
    if hdfs_user is None:
      raise ValueError('hdfs_user is not set')
    if not isinstance(self._full_urls, bool):
      raise ValueError(
          'hdfs_full_urls should be bool, got: %s', self._full_urls)
    self._hdfs_client = hdfs.InsecureClient(
        'http://%s:%s' % (hdfs_host, str(hdfs_port)), user=hdfs_user)

  @classmethod
  def scheme(cls):
    return 'hdfs'

  def _parse_url(self, url):
    """Verifies that url begins with hdfs:// prefix, strips it and adds a
    leading /.

    Parsing behavior is determined by HadoopFileSystemOptions.hdfs_full_urls.

    Args:
      url: (str) A URL in the form hdfs://path/...
        or in the form hdfs://server/path/...

View on GitHub (pinned to 12126d8942)