microsoft/qlib · error · ValueError

request error: {url}

Error message

request error: {url}

What it means

Raised by IndexCollectorUS._request_history_companies (scripts/data_collector/us_index/collector.py:170) when a POST to HISTORY_COMPANIES_URL (formatted with trade_date) returns non-200. This endpoint yields the index constituents as of a historical trade date; unlike 588 it is wrapped in @deco_retry, so the error only surfaces after the decorator's retry budget is exhausted. A pickled cache per trade_date short-circuits successful past fetches.

Source

Thrown at scripts/data_collector/us_index/collector.py:170

    def filter_df(self, df: pd.DataFrame) -> pd.DataFrame:
        if len(df) >= 100 and "Ticker" in df.columns:
            return df.loc[:, ["Ticker"]].copy()

    @property
    def bench_start_date(self) -> pd.Timestamp:
        return pd.Timestamp("2003-01-02")

    @deco_retry
    def _request_history_companies(self, trade_date: pd.Timestamp, use_cache: bool = True) -> pd.DataFrame:
        trade_date = trade_date.strftime("%Y-%m-%d")
        cache_path = self.cache_dir.joinpath(f"{trade_date}_history_companies.pkl")
        if cache_path.exists() and use_cache:
            df = pd.read_pickle(cache_path)
        else:
            url = self.HISTORY_COMPANIES_URL.format(trade_date=trade_date)
            resp = requests.post(url, timeout=None)
            if resp.status_code != 200:
                raise ValueError(f"request error: {url}")
            df = pd.DataFrame(resp.json()["aaData"])
            df[self.DATE_FIELD_NAME] = trade_date
            df.rename(columns={"Name": "name", "Symbol": self.SYMBOL_FIELD_NAME}, inplace=True)
            if not df.empty:
                df.to_pickle(cache_path)
        return df

    def get_history_companies(self):
        logger.info(f"start get history companies......")
        all_history = []
        error_list = []
        with tqdm(total=len(self.calendar_list)) as p_bar:
            with ThreadPoolExecutor(max_workers=self.MAX_WORKERS) as executor:
                for _trading_date, _df in zip(
                    self.calendar_list, executor.map(self._request_history_companies, self.calendar_list)
                ):
                    if _df.empty:
                        error_list.append(_trading_date)

View on GitHub (pinned to 79633dd950)

Solutions

1. Simply rerun get_history_companies: cached dates are skipped via <date>_history_companies.pkl, so only the failed dates refetch.
2. Narrow the date range (--start/--end) to dates the vendor actually covers.
3. Slow down the loop or run in smaller batches to stay under vendor rate limits.
4. If all dates fail, verify HISTORY_COMPANIES_URL with curl and update the constant if the endpoint moved.

Example fix

# before
python collector.py update_data_to_bin --index_name SP500 ...  # full range, throttled mid-run

# after
# rerun same command; cache skips completed dates
python collector.py update_data_to_bin --index_name SP500 ...

missing = [d for d in trade_dates if not (collector.cache_dir / f"{d:%Y-%m-%d}_history_companies.pkl").exists()]
print(f'{len(missing)} dates to fetch; expect vendor throttling on long backfills')

try:
    history = collector.get_history_companies()
except ValueError as e:
    if 'request error' in str(e):
        # rerun; per-date pickles skip already-fetched dates
        raise SystemExit('history endpoint throttled; rerun the same command to resume from cache')
    raise

When it happens

Trigger: Calling get_history_companies() (which iterates the calendar and POSTs per trade_date) when the history endpoint persistently returns non-200 for a date: dates the vendor does not cover, vendor blocking after many rapid POSTs (timeout=None means very slow responses also count against wall time, not status), or endpoint changes.

Common situations: Long backfills hammering the vendor until throttled; requesting trade dates before the vendor's history begins; stale HISTORY_COMPANIES_URL constant after a site redesign; partial runs where some dates cached and later ones fail.

Related errors

• request error: {self._target_url}
• Invalid mount path
• Failed to create directory {mount_path}, please create {moun
• nfs-common is not found, please install it by execute: sudo
• Mount failed: requires sudo or permission denied

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/4d10d7d390911f70. Report an issue: GitHub.