{"record":{"id":"e1df9eb1ba497b00","repo":"microsoft/qlib","slug":"request-error-self-target-url","errorCode":null,"errorMessage":"request error: {self._target_url}","messagePattern":"request error: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/data_collector/us_index/collector.py","lineNumber":119,"sourceCode":"    @property\n    def calendar_list(self) -> List[pd.Timestamp]:\n        \"\"\"get history trading date\n\n        Returns\n        -------\n            calendar list\n        \"\"\"\n        _calendar_list = getattr(self, \"_calendar_list\", None)\n        if _calendar_list is None:\n            _calendar_list = list(filter(lambda x: x >= self.bench_start_date, get_calendar_list(\"US_ALL\")))\n            setattr(self, \"_calendar_list\", _calendar_list)\n        return _calendar_list\n\n    def _request_new_companies(self) -> requests.Response:\n        headers = {\"User-Agent\": self._ua.random}\n        resp = requests.get(self._target_url, timeout=None, headers=headers)\n        if resp.status_code != 200:\n            raise ValueError(f\"request error: {self._target_url}\")\n\n        return resp\n\n    def set_default_date_range(self, df: pd.DataFrame) -> pd.DataFrame:\n        _df = df.copy()\n        _df[self.SYMBOL_FIELD_NAME] = _df[self.SYMBOL_FIELD_NAME].str.strip()\n        _df[self.START_DATE_FIELD] = self.bench_start_date\n        _df[self.END_DATE_FIELD] = self.DEFAULT_END_DATE\n        return _df.loc[:, self.INSTRUMENTS_COLUMNS]\n\n    def get_new_companies(self):\n        logger.info(f\"get new companies {self.index_name} ......\")\n        _data = deco_retry(retry=self._request_retry, retry_sleep=self._retry_sleep)(self._request_new_companies)()\n        df_list = pd.read_html(StringIO(_data.text))\n        for _df in df_list:\n            _df = self.filter_df(_df)\n            if (_df is not None) and (not _df.empty):\n                _df.columns = [self.SYMBOL_FIELD_NAME]","sourceCodeStart":101,"sourceCodeEnd":137,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/scripts/data_collector/us_index/collector.py#L101-L137","documentation":"Raised by IndexCollectorUS._request_new_companies (scripts/data_collector/us_index/collector.py:119) when requests.get(self._target_url) returns a non-200 status. This fetches the current constituents of the US index (e.g. SP500/NASDAQ100) from the vendor page; anything but 200 (403 anti-bot, 404 moved, 5xx, redirect-to-login) aborts before parsing.","triggerScenarios":"Calling get_new_companies()/save_new_companies on IndexCollectorUS when _target_url returns non-200: rate-limited or blocked scraper (note timeout=None, so it waits indefinitely for slow hosts), changed URL, or vendor outage. The random User-Agent header is already applied, so blocking usually means IP-level throttling.","commonSituations":"Running the US index collector repeatedly from one IP (CI jobs) and getting 403/429; the vendor page moving and HISTORY/COMPANIES URL constants becoming stale; proxy environments mangling the request.","solutions":["Retry after a delay — most non-200s here are transient throttling of scripted clients.","Open _target_url in a browser/curl to confirm it still resolves; if the vendor moved it, update the URL constant in the subclass.","Run from a different network/IP or reduce collection frequency to avoid anti-bot limits.","If a proxy is required, configure requests accordingly (HTTPS_PROXY env) so the status is not a proxy error page."],"exampleFix":"# before\nresp = requests.get(self._target_url, timeout=None, headers=headers)\nif resp.status_code != 200:\n    raise ValueError(f\"request error: {self._target_url}\")\n\n# after (bounded retry)\nimport time\nfor attempt in range(3):\n    resp = requests.get(self._target_url, timeout=30, headers=headers)\n    if resp.status_code == 200:\n        break\n    time.sleep(10 * (attempt + 1))\nelse:\n    raise ValueError(f\"request error after retries: {self._target_url} (last status {resp.status_code})\")","handlingStrategy":"retry","validationCode":"import requests\nresp = requests.get(collector._target_url, timeout=30, headers={'User-Agent': 'Mozilla/5.0'})\nif resp.status_code != 200:\n    raise SystemExit(f'index source unreachable (HTTP {resp.status_code}); retry later or change network')","typeGuard":null,"tryCatchPattern":"try:\n    companies = collector.get_new_companies()\nexcept ValueError as e:\n    if 'request error' in str(e):\n        time.sleep(120)\n        companies = collector.get_new_companies()  # one bounded retry\n    else:\n        raise","preventionTips":["Retry with backoff; anti-bot 403/429 responses are usually transient.","Vary IP or slow down when scraping index constituent pages in CI."],"tags":["us-index","http","anti-bot","upstream"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}