apache/beam · error · ValueError

Could not parse url

Error message

Could not parse url: %s

What it means

HadoopFileSystem._parse_url splits an HDFS path into server and path components using a regular expression whose choice depends on self._full_urls. If the URL does not match the expected 'hdfs://...' shape for the current mode, it raises ValueError 'Could not parse url: %s'. This method backs join, split, mkdirs, _list, create, and open, so any filesystem operation on a malformed path hits it.

Solutions

  1. Prefix the path with the scheme: 'hdfs://<host>:<port>/user/data/file.txt'.
  2. Make hdfs_full_urls consistent with the URL format you pass (False for bare '/path' form, True for full URLs).
  3. Normalize/validate URLs with re.match on the expected pattern before calling filesystem methods.
  4. Fix the scheme spelling and regenerate constructed paths (e.g. f'hdfs://{host}:{port}{path}').

Example fix

// before
with hdfs.open('/user/output/data.txt') as f: ...
// after
with hdfs.open('hdfs://namenode:50070/user/output/data.txt') as f: ...
Defensive patterns

Strategy: validation

Validate before calling

import re
_URL_RE = re.compile(r'hdfs://([^/]+)(/.*)?')
def is_hdfs_url(url: str, full: bool) -> bool:
    if full:
        return bool(re.match(r'hdfs://[^/]+(/.*)?', url))
    return url.startswith('/') or bool(_URL_RE.match(url))

Type guard

def looks_like_hdfs_path(x: object) -> bool:
    return isinstance(x, str) and x.startswith('hdfs://')

Try / catch

try:
    files = fs._list(url)
except ValueError as e:
    if str(e).startswith('Could not parse url'):
        url = f'hdfs://{host}:{port}{url}'
        files = fs._list(url)
    else:
        raise

Prevention

When it happens

Trigger: Calling HadoopFileSystem operations with a path lacking the 'hdfs://' scheme (e.g. '/user/data/file.txt') when _full_urls is False, or a malformed/full URL not matching _FULL_URL_RE when _full_urls is True.

Common situations: Mixing local-style paths with an HDFS filesystem; typos in the scheme ('hdfs://'); mismatch between the hdfs_full_urls setting and the URL form actually used; paths built by string concatenation losing the scheme.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7975c466efac9882. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:171

    Parsing behavior is determined by HadoopFileSystemOptions.hdfs_full_urls.

    Args:
      url: (str) A URL in the form hdfs://path/...
        or in the form hdfs://server/path/...

    Raises:
      ValueError if the URL doesn't match the expect format.

    Returns:
      (str, str) If using hdfs_full_urls, for an input of
      'hdfs://server/path/...' will return (server, '/path/...').
      Otherwise, for an input of 'hdfs://path/...', will return
      ('', '/path/...').
    """
    if not self._full_urls:
      m = _URL_RE.match(url)
      if m is None:
        raise ValueError('Could not parse url: %s' % url)
      return '', m.group(1)
    else:
      m = _FULL_URL_RE.match(url)
      if m is None:
        raise ValueError('Could not parse url: %s' % url)
      return m.group(1), m.group(2) or '/'

  def join(self, base_url, *paths):
    """Join two or more pathname components.

    Args:
      base_url: string path of the first component of the path.
        Must start with hdfs://.
      paths: path components to be added

    Returns:
      Full url after combining all the passed components.
    """

View on GitHub (pinned to 12126d8942)