{"record":{"id":"e76c4c5a94de4188","repo":"apache/beam","slug":"hdfs-port-is-not-set","errorCode":null,"errorMessage":"hdfs_port is not set","messagePattern":"hdfs_port is not set","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":136,"sourceCode":"    logging.getLogger('hdfs.client').setLevel(logging.WARN)\n    if pipeline_options is None:\n      raise ValueError('pipeline_options is not set')\n    if isinstance(pipeline_options, PipelineOptions):\n      hdfs_options = pipeline_options.view_as(HadoopFileSystemOptions)\n      hdfs_host = hdfs_options.hdfs_host\n      hdfs_port = hdfs_options.hdfs_port\n      hdfs_user = hdfs_options.hdfs_user\n      self._full_urls = hdfs_options.hdfs_full_urls\n    else:\n      hdfs_host = pipeline_options.get('hdfs_host')\n      hdfs_port = pipeline_options.get('hdfs_port')\n      hdfs_user = pipeline_options.get('hdfs_user')\n      self._full_urls = pipeline_options.get('hdfs_full_urls', False)\n\n    if hdfs_host is None:\n      raise ValueError('hdfs_host is not set')\n    if hdfs_port is None:\n      raise ValueError('hdfs_port is not set')\n    if hdfs_user is None:\n      raise ValueError('hdfs_user is not set')\n    if not isinstance(self._full_urls, bool):\n      raise ValueError(\n          'hdfs_full_urls should be bool, got: %s', self._full_urls)\n    self._hdfs_client = hdfs.InsecureClient(\n        'http://%s:%s' % (hdfs_host, str(hdfs_port)), user=hdfs_user)\n\n  @classmethod\n  def scheme(cls):\n    return 'hdfs'\n\n  def _parse_url(self, url):\n    \"\"\"Verifies that url begins with hdfs:// prefix, strips it and adds a\n    leading /.\n\n    Parsing behavior is determined by HadoopFileSystemOptions.hdfs_full_urls.\n","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L118-L154","documentation":"HadoopFileSystem validates that hdfs_port is provided when building its hdfs.InsecureClient. A missing hdfs_port raises ValueError 'hdfs_port is not set', since the client needs an HTTP port to reach the HDFS WebHDFS endpoint.","triggerScenarios":"Initializing HadoopFileSystem with options that include hdfs_host and hdfs_user but no 'hdfs_port' / --hdfs_port flag.","commonSituations":"Config copied from examples that omit the port; WebHDFS port (typically 50070/9870) unknown to the user; dict-based options missing the key.","solutions":["Add --hdfs_port=<port> (WebHDFS port, e.g. 50070 or 9870 for Hadoop 3.x) to the pipeline options.","Include 'hdfs_port': 50070 in the dict of options.","Confirm the correct WebHDFS port against your cluster's hdfs-site.xml (dfs.namenode.http-address).","Validate all of hdfs_host/hdfs_port/hdfs_user together before launching."],"exampleFix":"// before\nPipelineOptions(['--hdfs_host=namenode.example.com', '--hdfs_user=hduser'])\n// after\nPipelineOptions(['--hdfs_host=namenode.example.com', '--hdfs_port=50070', '--hdfs_user=hduser'])","handlingStrategy":"validation","validationCode":"def require_hdfs_port(opts):\n    v = (opts.get('hdfs_port') if isinstance(opts, dict)\n         else opts.view_as(HadoopFileSystemOptions).hdfs_port)\n    if not v:\n        raise ValueError('Set --hdfs_port (WebHDFS port, e.g. 50070) before launching')\n    return v","typeGuard":"None","tryCatchPattern":"try:\n    fs = HadoopFileSystem(pipeline_options=opts)\nexcept ValueError as e:\n    if 'hdfs_port is not set' in str(e):\n        raise SystemExit('Add --hdfs_port=<webhdfs-port> to pipeline options')\n    raise","preventionTips":["Confirm the WebHDFS port from dfs.namenode.http-address (50070 or 9870)","Always pass --hdfs_port with --hdfs_host","Validate all HDFS options together pre-launch","Avoid copy-pasting example configs without the port"],"tags":["hdfs","configuration","python","apache-beam"],"backgroundTag":"missing-required-config-field","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}