microsoft/qlib · error · ValueError

request error: {self._target_url}

Error message

request error: {self._target_url}

What it means

Raised by IndexCollectorUS._request_new_companies (scripts/data_collector/us_index/collector.py:119) when requests.get(self._target_url) returns a non-200 status. This fetches the current constituents of the US index (e.g. SP500/NASDAQ100) from the vendor page; anything but 200 (403 anti-bot, 404 moved, 5xx, redirect-to-login) aborts before parsing.

Source

Thrown at scripts/data_collector/us_index/collector.py:119

    @property
    def calendar_list(self) -> List[pd.Timestamp]:
        """get history trading date

        Returns
        -------
            calendar list
        """
        _calendar_list = getattr(self, "_calendar_list", None)
        if _calendar_list is None:
            _calendar_list = list(filter(lambda x: x >= self.bench_start_date, get_calendar_list("US_ALL")))
            setattr(self, "_calendar_list", _calendar_list)
        return _calendar_list

    def _request_new_companies(self) -> requests.Response:
        headers = {"User-Agent": self._ua.random}
        resp = requests.get(self._target_url, timeout=None, headers=headers)
        if resp.status_code != 200:
            raise ValueError(f"request error: {self._target_url}")

        return resp

    def set_default_date_range(self, df: pd.DataFrame) -> pd.DataFrame:
        _df = df.copy()
        _df[self.SYMBOL_FIELD_NAME] = _df[self.SYMBOL_FIELD_NAME].str.strip()
        _df[self.START_DATE_FIELD] = self.bench_start_date
        _df[self.END_DATE_FIELD] = self.DEFAULT_END_DATE
        return _df.loc[:, self.INSTRUMENTS_COLUMNS]

    def get_new_companies(self):
        logger.info(f"get new companies {self.index_name} ......")
        _data = deco_retry(retry=self._request_retry, retry_sleep=self._retry_sleep)(self._request_new_companies)()
        df_list = pd.read_html(StringIO(_data.text))
        for _df in df_list:
            _df = self.filter_df(_df)
            if (_df is not None) and (not _df.empty):
                _df.columns = [self.SYMBOL_FIELD_NAME]

View on GitHub (pinned to 79633dd950)

Solutions

1. Retry after a delay — most non-200s here are transient throttling of scripted clients.
2. Open _target_url in a browser/curl to confirm it still resolves; if the vendor moved it, update the URL constant in the subclass.
3. Run from a different network/IP or reduce collection frequency to avoid anti-bot limits.
4. If a proxy is required, configure requests accordingly (HTTPS_PROXY env) so the status is not a proxy error page.

Example fix

# before
resp = requests.get(self._target_url, timeout=None, headers=headers)
if resp.status_code != 200:
    raise ValueError(f"request error: {self._target_url}")

# after (bounded retry)
import time
for attempt in range(3):
    resp = requests.get(self._target_url, timeout=30, headers=headers)
    if resp.status_code == 200:
        break
    time.sleep(10 * (attempt + 1))
else:
    raise ValueError(f"request error after retries: {self._target_url} (last status {resp.status_code})")

import requests
resp = requests.get(collector._target_url, timeout=30, headers={'User-Agent': 'Mozilla/5.0'})
if resp.status_code != 200:
    raise SystemExit(f'index source unreachable (HTTP {resp.status_code}); retry later or change network')

try:
    companies = collector.get_new_companies()
except ValueError as e:
    if 'request error' in str(e):
        time.sleep(120)
        companies = collector.get_new_companies()  # one bounded retry
    else:
        raise

When it happens

Trigger: Calling get_new_companies()/save_new_companies on IndexCollectorUS when _target_url returns non-200: rate-limited or blocked scraper (note timeout=None, so it waits indefinitely for slow hosts), changed URL, or vendor outage. The random User-Agent header is already applied, so blocking usually means IP-level throttling.

Common situations: Running the US index collector repeatedly from one IP (CI jobs) and getting 403/429; the vendor page moving and HISTORY/COMPANIES URL constants becoming stale; proxy environments mangling the request.

Related errors

• request error: {url}
• response status: {_status}, url={url}
• get new companies error: {self.index_name}

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/e1df9eb1ba497b00. Report an issue: GitHub.