{"record":{"id":"f9be43ce6a55f22c","repo":"microsoft/qlib","slug":"the-complete-list-of-stocks-is-not-available","errorCode":null,"errorMessage":"The complete list of stocks is not available.","messagePattern":"The complete list of stocks is not available\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/data_collector/utils.py","lineNumber":256,"sourceCode":"                logger.info(\n                    f\"Page {page}: fetch {len(current_symbols)} stocks:[{current_symbols[0]} ... {current_symbols[-1]}]\"\n                )\n\n                page += 1\n\n                # sleep time to avoid overloading the server\n                time.sleep(0.5)\n\n            except requests.exceptions.HTTPError as e:\n                raise requests.exceptions.HTTPError(\n                    f\"Request to {base_url} failed with status code {resp.status_code}\"\n                ) from e\n            except Exception as e:\n                logger.warning(\"An error occurred while extracting data from the response.\")\n                raise\n\n        if len(_symbols) < 3900:\n            raise ValueError(\"The complete list of stocks is not available.\")\n\n        # Add suffix after the stock code to conform to yahooquery standard, otherwise the data will not be fetched.\n        _symbols = [\n            _symbol + \".ss\" if _symbol.startswith(\"6\") else _symbol + \".sz\" if _symbol.startswith((\"0\", \"3\")) else None\n            for _symbol in _symbols\n        ]\n        _symbols = [_symbol for _symbol in _symbols if _symbol is not None]\n\n        return set(_symbols)\n\n    if _HS_SYMBOLS is None:\n        symbols = set()\n        _retry = 60\n        # It may take multiple times to get the complete\n        while len(symbols) < MINIMUM_SYMBOLS_NUM:\n            symbols |= _get_symbol()\n            time.sleep(3)\n","sourceCodeStart":238,"sourceCodeEnd":274,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/scripts/data_collector/utils.py#L238-L274","documentation":"Raised in get_hs_symbols (scripts/data_collector/utils.py:256) when the paginated eastmoney scraper (99.push2.eastmoney.com/api/qt/clist/get) collects fewer than 3900 A-share symbols (MINIMUM_SYMBOLS_NUM). The threshold is a completeness check: the loop breaks on the first invalid/empty page, so a truncated scrape (rate limit, partial JSON, changed response shape) silently under-collects and this error catches it before suffixing .ss/.sz.","triggerScenarios":"Calling get_hs_symbols() when any page returns an unexpected structure (data/diff missing → 'Invalid response structure' break), an early empty page, or an HTTP error mid-pagination — leaving len(_symbols) below 3900. Also triggers if the A-share universe itself shifts below the hard-coded threshold.","commonSituations":"Being rate-limited by eastmoney during the 0.5s-interval page loop; eastmoney changing its JSON shape (key renames) so every page looks 'invalid'; newer environments where the expected symbol count assumption is stale.","solutions":["Retry the call after a pause — transient throttling during pagination is the top cause.","Inspect logged 'Invalid response structure on page N' warnings to see which page broke, then curl that page manually to compare the JSON shape with the parser (data.data.diff[].f12).","If eastmoney changed its API, update the params/keys in the scraper (fs/fields values) to match the current endpoint.","If the market genuinely has fewer listings than 3900 in your filter set, adjust MINIMUM_SYMBOLS_NUM accordingly."],"exampleFix":"# before\nsymbols = get_hs_symbols()  # raises when scrape truncated\n\n# after\nfrom scripts.data_collector.utils import get_hs_symbols\nimport time\nfor attempt in range(3):\n    try:\n        symbols = get_hs_symbols()\n        break\n    except ValueError:\n        time.sleep(30 * (attempt + 1))\nelse:\n    raise RuntimeError('eastmoney symbol scrape incomplete after retries')","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    symbols = get_hs_symbols()\nexcept ValueError as e:\n    if 'complete list' in str(e):\n        time.sleep(60)\n        symbols = get_hs_symbols()  # one more attempt after backoff\n    else:\n        raise","preventionTips":["Watch for 'Invalid response structure on page N' warnings — they explain the truncated count.","Keep the pagination sleep(0.5) intact; shortening it invites throttling and partial scrapes."],"tags":["eastmoney","cn-symbols","pagination","completeness-check"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}