{"record":{"id":"7a3c0bdb79d09d3d","repo":"HKUDS/Vibe-Trading","slug":"sec-fails-to-deliver-index-listed-no-archives","errorCode":null,"errorMessage":"SEC fails-to-deliver index listed no archives","messagePattern":"SEC fails-to-deliver index listed no archives","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"agent/src/tools/institutional_holdings_tool.py","lineNumber":448,"sourceCode":"    (``/files/data/fails-deliver-data/`` and ``/files/data/other/fails-deliver-data/``)\n    and only the page knows which is which.\n\n    Returns:\n        Newest-first absolute ``.zip`` URLs, at most the configured file count.\n\n    Raises:\n        requests.RequestException: If the landing page cannot be fetched.\n        RuntimeError: If the page lists no archives.\n    \"\"\"\n    data_config = get_env_config().data\n    pinned = data_config.vibe_trading_sec_ftd_url.strip()\n    if pinned:\n        return [pinned]\n    wanted = max(1, min(data_config.vibe_trading_sec_ftd_files, _MAX_FTD_FILES))\n    html = _sec_get(_FTD_INDEX_URL).text\n    hrefs = re.findall(r'href=\"(/[^\"]*cnsfails\\d{6}[ab]\\.zip)\"', html)\n    if not hrefs:\n        raise RuntimeError(\"SEC fails-to-deliver index listed no archives\")\n    ordered = sorted(dict.fromkeys(hrefs), key=lambda h: h.rsplit(\"/\", 1)[-1], reverse=True)\n    return [f\"https://www.sec.gov{h}\" for h in ordered[:wanted]]\n\n\ndef _load_ftd_map(url: str, into: Dict[str, Tuple[str, str]]) -> None:\n    \"\"\"Merge one fails-to-deliver archive into ``into`` as ticker -> (cusip, description).\n\n    The pipe-delimited payload is ``SETTLEMENT DATE|CUSIP|SYMBOL|QUANTITY\n    (FAILS)|DESCRIPTION|PRICE``. Earlier entries win, so a newer archive passed\n    first keeps precedence over an older one.\n\n    Args:\n        url: Absolute URL of a ``cnsfailsYYYYMM{a,b}.zip`` archive.\n        into: Accumulator mutated in place.\n\n    Raises:\n        requests.RequestException: On transport failure.\n        zipfile.BadZipFile: If the payload is not a zip archive.","sourceCodeStart":430,"sourceCodeEnd":466,"githubUrl":"https://github.com/HKUDS/Vibe-Trading/blob/80ffdda44c5c4db0dd84d70e051cca591cea67df/agent/src/tools/institutional_holdings_tool.py#L430-L466","documentation":"_ftd_urls scrapes the SEC fails-to-deliver FOIA index page and regex-extracts links to cnsfails*.zip archives. If the regex finds nothing — empty index, moved page, or HTML schema change — this RuntimeError fires before any download. A pinned URL bypasses the scrape entirely.","triggerScenarios":"SEC restructures the FTD index page so hrefs no longer match the pattern; index temporarily empty; request returned an error/HTML block page. Called via _cusip_map.","commonSituations":"SEC site redesigns, bot-blocking returning an empty body, or network proxies stripping content.","solutions":["Configure the pinned FTD URL setting (vibe_trading config data_config) so the index is not scraped","Retry later — transient SEC issues are common","Update the href regex in _ftd_urls if the archive naming changed"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    urls = _ftd_urls()\nexcept RuntimeError:\n    urls = [PINNED_FTD_URL]  # configured fallback archive","preventionTips":["Configure the pinned FTD URL so the index scrape is skipped","Cache the discovered archive list","Add a canary test against the index page to catch schema drift early"],"tags":["sec-edger","web-scraping","http","config"],"backgroundTag":"upstream-page-scrape-failed","analyzedSha":"80ffdda44c5c4db0dd84d70e051cca591cea67df","analyzedAt":"2026-08-28T12:46:38.989Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}