{"record":{"id":"4d10d7d390911f70","repo":"microsoft/qlib","slug":"request-error-url","errorCode":null,"errorMessage":"request error: {url}","messagePattern":"request error: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/data_collector/us_index/collector.py","lineNumber":170,"sourceCode":"    def filter_df(self, df: pd.DataFrame) -> pd.DataFrame:\n        if len(df) >= 100 and \"Ticker\" in df.columns:\n            return df.loc[:, [\"Ticker\"]].copy()\n\n    @property\n    def bench_start_date(self) -> pd.Timestamp:\n        return pd.Timestamp(\"2003-01-02\")\n\n    @deco_retry\n    def _request_history_companies(self, trade_date: pd.Timestamp, use_cache: bool = True) -> pd.DataFrame:\n        trade_date = trade_date.strftime(\"%Y-%m-%d\")\n        cache_path = self.cache_dir.joinpath(f\"{trade_date}_history_companies.pkl\")\n        if cache_path.exists() and use_cache:\n            df = pd.read_pickle(cache_path)\n        else:\n            url = self.HISTORY_COMPANIES_URL.format(trade_date=trade_date)\n            resp = requests.post(url, timeout=None)\n            if resp.status_code != 200:\n                raise ValueError(f\"request error: {url}\")\n            df = pd.DataFrame(resp.json()[\"aaData\"])\n            df[self.DATE_FIELD_NAME] = trade_date\n            df.rename(columns={\"Name\": \"name\", \"Symbol\": self.SYMBOL_FIELD_NAME}, inplace=True)\n            if not df.empty:\n                df.to_pickle(cache_path)\n        return df\n\n    def get_history_companies(self):\n        logger.info(f\"start get history companies......\")\n        all_history = []\n        error_list = []\n        with tqdm(total=len(self.calendar_list)) as p_bar:\n            with ThreadPoolExecutor(max_workers=self.MAX_WORKERS) as executor:\n                for _trading_date, _df in zip(\n                    self.calendar_list, executor.map(self._request_history_companies, self.calendar_list)\n                ):\n                    if _df.empty:\n                        error_list.append(_trading_date)","sourceCodeStart":152,"sourceCodeEnd":188,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/scripts/data_collector/us_index/collector.py#L152-L188","documentation":"Raised by IndexCollectorUS._request_history_companies (scripts/data_collector/us_index/collector.py:170) when a POST to HISTORY_COMPANIES_URL (formatted with trade_date) returns non-200. This endpoint yields the index constituents as of a historical trade date; unlike 588 it is wrapped in @deco_retry, so the error only surfaces after the decorator's retry budget is exhausted. A pickled cache per trade_date short-circuits successful past fetches.","triggerScenarios":"Calling get_history_companies() (which iterates the calendar and POSTs per trade_date) when the history endpoint persistently returns non-200 for a date: dates the vendor does not cover, vendor blocking after many rapid POSTs (timeout=None means very slow responses also count against wall time, not status), or endpoint changes.","commonSituations":"Long backfills hammering the vendor until throttled; requesting trade dates before the vendor's history begins; stale HISTORY_COMPANIES_URL constant after a site redesign; partial runs where some dates cached and later ones fail.","solutions":["Simply rerun get_history_companies: cached dates are skipped via <date>_history_companies.pkl, so only the failed dates refetch.","Narrow the date range (--start/--end) to dates the vendor actually covers.","Slow down the loop or run in smaller batches to stay under vendor rate limits.","If all dates fail, verify HISTORY_COMPANIES_URL with curl and update the constant if the endpoint moved."],"exampleFix":"# before\npython collector.py update_data_to_bin --index_name SP500 ...  # full range, throttled mid-run\n\n# after\n# rerun same command; cache skips completed dates\npython collector.py update_data_to_bin --index_name SP500 ...","handlingStrategy":"retry","validationCode":"missing = [d for d in trade_dates if not (collector.cache_dir / f\"{d:%Y-%m-%d}_history_companies.pkl\").exists()]\nprint(f'{len(missing)} dates to fetch; expect vendor throttling on long backfills')","typeGuard":null,"tryCatchPattern":"try:\n    history = collector.get_history_companies()\nexcept ValueError as e:\n    if 'request error' in str(e):\n        # rerun; per-date pickles skip already-fetched dates\n        raise SystemExit('history endpoint throttled; rerun the same command to resume from cache')\n    raise","preventionTips":["Exploit the per-date pickle cache: simply rerun after throttling to resume.","Restrict date ranges to what the vendor actually covers."],"tags":["us-index","http","retry","cache","history"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}