microsoft/qlib · error · ValueError

The complete list of stocks is not available.

Error message

The complete list of stocks is not available.

What it means

Raised in get_hs_symbols (scripts/data_collector/utils.py:256) when the paginated eastmoney scraper (99.push2.eastmoney.com/api/qt/clist/get) collects fewer than 3900 A-share symbols (MINIMUM_SYMBOLS_NUM). The threshold is a completeness check: the loop breaks on the first invalid/empty page, so a truncated scrape (rate limit, partial JSON, changed response shape) silently under-collects and this error catches it before suffixing .ss/.sz.

Source

Thrown at scripts/data_collector/utils.py:256

                logger.info(
                    f"Page {page}: fetch {len(current_symbols)} stocks:[{current_symbols[0]} ... {current_symbols[-1]}]"
                )

                page += 1

                # sleep time to avoid overloading the server
                time.sleep(0.5)

            except requests.exceptions.HTTPError as e:
                raise requests.exceptions.HTTPError(
                    f"Request to {base_url} failed with status code {resp.status_code}"
                ) from e
            except Exception as e:
                logger.warning("An error occurred while extracting data from the response.")
                raise

        if len(_symbols) < 3900:
            raise ValueError("The complete list of stocks is not available.")

        # Add suffix after the stock code to conform to yahooquery standard, otherwise the data will not be fetched.
        _symbols = [
            _symbol + ".ss" if _symbol.startswith("6") else _symbol + ".sz" if _symbol.startswith(("0", "3")) else None
            for _symbol in _symbols
        ]
        _symbols = [_symbol for _symbol in _symbols if _symbol is not None]

        return set(_symbols)

    if _HS_SYMBOLS is None:
        symbols = set()
        _retry = 60
        # It may take multiple times to get the complete
        while len(symbols) < MINIMUM_SYMBOLS_NUM:
            symbols |= _get_symbol()
            time.sleep(3)

View on GitHub (pinned to 79633dd950)

Solutions

1. Retry the call after a pause — transient throttling during pagination is the top cause.
2. Inspect logged 'Invalid response structure on page N' warnings to see which page broke, then curl that page manually to compare the JSON shape with the parser (data.data.diff[].f12).
3. If eastmoney changed its API, update the params/keys in the scraper (fs/fields values) to match the current endpoint.
4. If the market genuinely has fewer listings than 3900 in your filter set, adjust MINIMUM_SYMBOLS_NUM accordingly.

Example fix

# before
symbols = get_hs_symbols()  # raises when scrape truncated

# after
from scripts.data_collector.utils import get_hs_symbols
import time
for attempt in range(3):
    try:
        symbols = get_hs_symbols()
        break
    except ValueError:
        time.sleep(30 * (attempt + 1))
else:
    raise RuntimeError('eastmoney symbol scrape incomplete after retries')

try:
    symbols = get_hs_symbols()
except ValueError as e:
    if 'complete list' in str(e):
        time.sleep(60)
        symbols = get_hs_symbols()  # one more attempt after backoff
    else:
        raise

When it happens

Trigger: Calling get_hs_symbols() when any page returns an unexpected structure (data/diff missing → 'Invalid response structure' break), an early empty page, or an HTTP error mid-pagination — leaving len(_symbols) below 3900. Also triggers if the A-share universe itself shifts below the hard-coded threshold.

Common situations: Being rate-limited by eastmoney during the 0.5s-interval page loop; eastmoney changing its JSON shape (key renames) so every page looks 'invalid'; newer environments where the expected symbol count assumption is stale.

Related errors

• request error
• request error
• The fund contains 每*份收益

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/f9be43ce6a55f22c. Report an issue: GitHub.