microsoft/qlib · error · ValueError

get new companies error: {self.index_name}

Error message

get new companies error: {self.index_name}

What it means

Raised by IndexCollector.save_new_companies (scripts/data_collector/index.py:143) when get_new_companies() returns None or an empty DataFrame for the given index_name. The collector needs the current constituent list as the seed for computing history; an empty response almost always means the upstream request for constituents failed or was blocked rather than that the index truly has no members.

Source

Thrown at scripts/data_collector/index.py:143

        inst_df: pd.DataFrame
            inst_df.columns = [self.SYMBOL_FIELD_NAME, self.START_DATE_FIELD, self.END_DATE_FIELD]

        Returns
        -------

        """
        raise NotImplementedError("rewrite format_datetime")

    def save_new_companies(self):
        """save new companies

        Examples
        -------
            $ python collector.py save_new_companies --index_name CSI300 --qlib_dir ~/.qlib/qlib_data/cn_data
        """
        df = self.get_new_companies()
        if df is None or df.empty:
            raise ValueError(f"get new companies error: {self.index_name}")
        df = df.drop_duplicates([self.SYMBOL_FIELD_NAME])
        df.loc[:, self.INSTRUMENTS_COLUMNS].to_csv(
            self.instruments_dir.joinpath(f"{self.index_name.lower()}_only_new.txt"), sep="\t", index=False, header=None
        )

    def get_changes_with_history_companies(self, history_companies: pd.DataFrame) -> pd.DataFrame:
        """get changes with history companies

        Parameters
        ----------
        history_companies : pd.DataFrame
            symbol        date
            SH600000   2020-11-11

            dtypes:
                symbol: str
                date: pd.Timestamp

View on GitHub (pinned to 79633dd950)

Solutions

1. Retry the command; transient upstream blocking (rate limits) is the most common cause.
2. Verify --index_name is one the collector supports and that its request URL resolves (open it in a browser / curl it).
3. Check the subclass's get_new_companies/_request_new_companies parsing against the current upstream page format; fix the parser if the site changed.
4. Set a custom User-Agent / add delays if the source is anti-bot filtering the default client.

Example fix

# before
subprocess.run(['python', 'index.py', 'save_new_companies', '--index_name', 'CSI300', '--qlib_dir', qlib_dir], check=True)

# after (probe before running)
df = collector.get_new_companies()
if df is None or df.empty:
    raise SystemExit('upstream returned no constituents; check index_name/network, then retry')
collector.save_new_companies()

df = collector.get_new_companies()
if df is None or df.empty:
    raise SystemExit(f'no constituents fetched for {collector.index_name}; check network/upstream, then retry')
collector.save_new_companies()

try:
    collector.save_new_companies()
except ValueError as e:
    if 'get new companies error' in str(e):
        time.sleep(60); collector.save_new_companies()  # retry once after backoff
    else:
        raise

When it happens

Trigger: Running `python index.py save_new_companies --index_name CSI300 --qlib_dir ...` when the per-index source (e.g. CSIndex/CSI download endpoint, or the US collector's _request_new_companies) returns an empty/failed payload — non-200 mapped to empty, anti-bot blocking, or an unsupported index_name that matches no data.

Common situations: CSI website blocking scripted downloads (User-Agent/rate limits); misspelled or unsupported --index_name; upstream layout change breaking the parsing so the DataFrame ends up empty.

Related errors

• request error: {self._target_url}

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/ddb67edb62c2037e. Report an issue: GitHub.