{"record":{"id":"ddb67edb62c2037e","repo":"microsoft/qlib","slug":"get-new-companies-error-self-index-name","errorCode":null,"errorMessage":"get new companies error: {self.index_name}","messagePattern":"get new companies error: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/data_collector/index.py","lineNumber":143,"sourceCode":"        inst_df: pd.DataFrame\n            inst_df.columns = [self.SYMBOL_FIELD_NAME, self.START_DATE_FIELD, self.END_DATE_FIELD]\n\n        Returns\n        -------\n\n        \"\"\"\n        raise NotImplementedError(\"rewrite format_datetime\")\n\n    def save_new_companies(self):\n        \"\"\"save new companies\n\n        Examples\n        -------\n            $ python collector.py save_new_companies --index_name CSI300 --qlib_dir ~/.qlib/qlib_data/cn_data\n        \"\"\"\n        df = self.get_new_companies()\n        if df is None or df.empty:\n            raise ValueError(f\"get new companies error: {self.index_name}\")\n        df = df.drop_duplicates([self.SYMBOL_FIELD_NAME])\n        df.loc[:, self.INSTRUMENTS_COLUMNS].to_csv(\n            self.instruments_dir.joinpath(f\"{self.index_name.lower()}_only_new.txt\"), sep=\"\\t\", index=False, header=None\n        )\n\n    def get_changes_with_history_companies(self, history_companies: pd.DataFrame) -> pd.DataFrame:\n        \"\"\"get changes with history companies\n\n        Parameters\n        ----------\n        history_companies : pd.DataFrame\n            symbol        date\n            SH600000   2020-11-11\n\n            dtypes:\n                symbol: str\n                date: pd.Timestamp\n","sourceCodeStart":125,"sourceCodeEnd":161,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/scripts/data_collector/index.py#L125-L161","documentation":"Raised by IndexCollector.save_new_companies (scripts/data_collector/index.py:143) when get_new_companies() returns None or an empty DataFrame for the given index_name. The collector needs the current constituent list as the seed for computing history; an empty response almost always means the upstream request for constituents failed or was blocked rather than that the index truly has no members.","triggerScenarios":"Running `python index.py save_new_companies --index_name CSI300 --qlib_dir ...` when the per-index source (e.g. CSIndex/CSI download endpoint, or the US collector's _request_new_companies) returns an empty/failed payload — non-200 mapped to empty, anti-bot blocking, or an unsupported index_name that matches no data.","commonSituations":"CSI website blocking scripted downloads (User-Agent/rate limits); misspelled or unsupported --index_name; upstream layout change breaking the parsing so the DataFrame ends up empty.","solutions":["Retry the command; transient upstream blocking (rate limits) is the most common cause.","Verify --index_name is one the collector supports and that its request URL resolves (open it in a browser / curl it).","Check the subclass's get_new_companies/_request_new_companies parsing against the current upstream page format; fix the parser if the site changed.","Set a custom User-Agent / add delays if the source is anti-bot filtering the default client."],"exampleFix":"# before\nsubprocess.run(['python', 'index.py', 'save_new_companies', '--index_name', 'CSI300', '--qlib_dir', qlib_dir], check=True)\n\n# after (probe before running)\ndf = collector.get_new_companies()\nif df is None or df.empty:\n    raise SystemExit('upstream returned no constituents; check index_name/network, then retry')\ncollector.save_new_companies()","handlingStrategy":"validation","validationCode":"df = collector.get_new_companies()\nif df is None or df.empty:\n    raise SystemExit(f'no constituents fetched for {collector.index_name}; check network/upstream, then retry')\ncollector.save_new_companies()","typeGuard":null,"tryCatchPattern":"try:\n    collector.save_new_companies()\nexcept ValueError as e:\n    if 'get new companies error' in str(e):\n        time.sleep(60); collector.save_new_companies()  # retry once after backoff\n    else:\n        raise","preventionTips":["Probe get_new_companies() before running dependent steps.","Run constituent fetches off-peak and with delays to avoid upstream anti-bot blocks."],"tags":["index-collector","empty-response","upstream","constituents"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}