{"record":{"id":"7975c466efac9882","repo":"apache/beam","slug":"could-not-parse-url-s","errorCode":null,"errorMessage":"Could not parse url: %s","messagePattern":"Could not parse url: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":171,"sourceCode":"    Parsing behavior is determined by HadoopFileSystemOptions.hdfs_full_urls.\n\n    Args:\n      url: (str) A URL in the form hdfs://path/...\n        or in the form hdfs://server/path/...\n\n    Raises:\n      ValueError if the URL doesn't match the expect format.\n\n    Returns:\n      (str, str) If using hdfs_full_urls, for an input of\n      'hdfs://server/path/...' will return (server, '/path/...').\n      Otherwise, for an input of 'hdfs://path/...', will return\n      ('', '/path/...').\n    \"\"\"\n    if not self._full_urls:\n      m = _URL_RE.match(url)\n      if m is None:\n        raise ValueError('Could not parse url: %s' % url)\n      return '', m.group(1)\n    else:\n      m = _FULL_URL_RE.match(url)\n      if m is None:\n        raise ValueError('Could not parse url: %s' % url)\n      return m.group(1), m.group(2) or '/'\n\n  def join(self, base_url, *paths):\n    \"\"\"Join two or more pathname components.\n\n    Args:\n      base_url: string path of the first component of the path.\n        Must start with hdfs://.\n      paths: path components to be added\n\n    Returns:\n      Full url after combining all the passed components.\n    \"\"\"","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L153-L189","documentation":"HadoopFileSystem._parse_url splits an HDFS path into server and path components using a regular expression whose choice depends on self._full_urls. If the URL does not match the expected 'hdfs://...' shape for the current mode, it raises ValueError 'Could not parse url: %s'. This method backs join, split, mkdirs, _list, create, and open, so any filesystem operation on a malformed path hits it.","triggerScenarios":"Calling HadoopFileSystem operations with a path lacking the 'hdfs://' scheme (e.g. '/user/data/file.txt') when _full_urls is False, or a malformed/full URL not matching _FULL_URL_RE when _full_urls is True.","commonSituations":"Mixing local-style paths with an HDFS filesystem; typos in the scheme ('hdfs://'); mismatch between the hdfs_full_urls setting and the URL form actually used; paths built by string concatenation losing the scheme.","solutions":["Prefix the path with the scheme: 'hdfs://<host>:<port>/user/data/file.txt'.","Make hdfs_full_urls consistent with the URL format you pass (False for bare '/path' form, True for full URLs).","Normalize/validate URLs with re.match on the expected pattern before calling filesystem methods.","Fix the scheme spelling and regenerate constructed paths (e.g. f'hdfs://{host}:{port}{path}')."],"exampleFix":"// before\nwith hdfs.open('/user/output/data.txt') as f: ...\n// after\nwith hdfs.open('hdfs://namenode:50070/user/output/data.txt') as f: ...","handlingStrategy":"validation","validationCode":"import re\n_URL_RE = re.compile(r'hdfs://([^/]+)(/.*)?')\ndef is_hdfs_url(url: str, full: bool) -> bool:\n    if full:\n        return bool(re.match(r'hdfs://[^/]+(/.*)?', url))\n    return url.startswith('/') or bool(_URL_RE.match(url))","typeGuard":"def looks_like_hdfs_path(x: object) -> bool:\n    return isinstance(x, str) and x.startswith('hdfs://')","tryCatchPattern":"try:\n    files = fs._list(url)\nexcept ValueError as e:\n    if str(e).startswith('Could not parse url'):\n        url = f'hdfs://{host}:{port}{url}'\n        files = fs._list(url)\n    else:\n        raise","preventionTips":["Always include the hdfs:// scheme and host:port in paths passed to HadoopFileSystem","Keep hdfs_full_urls consistent with the URL format you actually use","Build paths via fs.join instead of string concatenation","Validate paths with a regex before filesystem calls"],"tags":["hdfs","url-parsing","python","apache-beam"],"backgroundTag":"invalid-url-format","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}