{"record":{"id":"e1f4b443070e8d2c","repo":"datawhalechina/hello-agents","slug":"arxiv-api-response-status","errorCode":null,"errorMessage":"ArXiv API请求失败: {response.status}","messagePattern":"ArXiv API请求失败: (.+?)","errorType":"exception","errorClass":"ExternalAPIException","httpStatus":500,"severity":"error","filePath":"Co-creation-projects/Apricity-InnocoreAI/agents/hunter.py","lineNumber":122,"sourceCode":"        # 添加时间过滤\n        date_filter = \"\"\n        if days_back > 0:\n            start_date = (datetime.now() - timedelta(days=days_back)).strftime(\"%Y%m%d\")\n            date_filter = f\"submittedDate:[{start_filter}0000 TO {datetime.now().strftime('%Y%m%d')}2359]\"\n        \n        params = {\n            \"search_query\": query,\n            \"start\": 0,\n            \"max_results\": max_papers * 2,  # 获取更多结果以便筛选\n            \"sortBy\": \"submittedDate\",\n            \"sortOrder\": \"descending\"\n        }\n        \n        try:\n            async with aiohttp.ClientSession() as session:\n                async with session.get(self.arxiv_base_url, params=params) as response:\n                    if response.status != 200:\n                        raise ExternalAPIException(f\"ArXiv API请求失败: {response.status}\")\n                    \n                    xml_content = await response.text()\n                    feed = feedparser.parse(xml_content)\n                    \n                    for entry in feed.entries:\n                        paper = {\n                            \"id\": entry.id.split(\"/\")[-1],\n                            \"title\": entry.title,\n                            \"authors\": [author.name for author in entry.authors],\n                            \"abstract\": entry.summary,\n                            \"published\": entry.published,\n                            \"pdf_url\": entry.link.replace('/abs/', '/pdf/') + '.pdf',\n                            \"source\": \"arxiv\",\n                            \"doi\": entry.get('arxiv_doi', ''),\n                            \"categories\": [tag.term for tag in entry.tags]\n                        }\n                        \n                        papers.append(paper)","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/datawhalechina/hello-agents/blob/606a07d341a47be773fab7f4b71177f53f96b2c3/Co-creation-projects/Apricity-InnocoreAI/agents/hunter.py#L104-L140","documentation":"ExternalAPIException raised in HunterAgent._search_papers_from_arxiv when the GET to arxiv_base_url (export.arxiv.org/api/query) returns any status other than 200, with the status code in the message. arXiv's public API has no key; non-200 almost always means rate limiting (429), temporary unavailability (503), or a malformed/oversized query string.","triggerScenarios":"Burst searches (max_results doubled by the code to max_papers*2) from one IP exceeding arXiv's ~1 req/3s courtesy limit -> 429; long multi-keyword query string with unencoded operators returning 400; arXiv maintenance window returning 5xx; corporate proxy intercepting with 403.","commonSituations":"Batch jobs searching dozens of keyword sets back-to-back; special characters (quotes, AND/OR braces) in keywords not URL-encoded by params; running hunter from cloud IPs that arXiv throttles harder.","solutions":["On 429/503: back off (sleep 3-10s, exponential) and retry the request a few times before surfacing the error.","Simplify/encode the search_query: quote keywords, limit boolean nesting, keep max_results ≤ 100 per call and page with 'start'.","Cache arXiv responses (same query within N minutes) so repeated runs don't re-hit the API.","Run from a stable IP or route through a polite cache (e.g. arXiv-compatible mirrors) when doing bulk harvests.","Check https://status.arxiv.org for outages when 5xx persists."],"exampleFix":"# before\nasync with session.get(self.arxiv_base_url, params=params) as response:\n    if response.status != 200:\n        raise ExternalAPIException(f\"ArXiv API请求失败: {response.status}\")\n\n# after — honor Retry-After / back off on throttling\nimport asyncio\nfor attempt in range(4):\n    async with session.get(self.arxiv_base_url, params=params) as response:\n        if response.status == 200:\n            xml_content = await response.text()\n            break\n        if response.status in (429, 500, 503) and attempt < 3:\n            retry_after = int(response.headers.get(\"Retry-After\", 3 * (attempt + 1)))\n            await asyncio.sleep(retry_after)\n            continue\n        raise ExternalAPIException(f\"ArXiv API请求失败: {response.status}\")","handlingStrategy":"retry","validationCode":"# Sanity-check the query before it leaves\nfrom urllib.parse import quote\nassert len(keywords) > 0, \"keywords required\"\nquery = \" AND \".join(f\"all:{quote(k)}\" for k in keywords)\nassert len(query) < 300, \"arXiv search_query too long; split into batches\"","typeGuard":null,"tryCatchPattern":"from agents.exceptions import ExternalAPIException\ntry:\n    papers = await hunter._search_papers_from_arxiv(kws, max_papers, days_back)\nexcept ExternalAPIException as e:\n    if any(code in str(e) for code in (\"429\", \"500\", \"503\")):\n        await asyncio.sleep(backoff())\n        papers = await hunter._search_papers_from_arxiv(kws, max_papers, days_back)\n    else:\n        raise","preventionTips":["Rate-limit arXiv calls to ~1 per 3 seconds and cache identical queries.","URL-encode keywords and keep search_query short and simply structured.","Honor Retry-After headers on 429 responses."],"tags":["python","arxiv","http","rate-limit","network","web-scraping"],"backgroundTag":null,"analyzedSha":"606a07d341a47be773fab7f4b71177f53f96b2c3","analyzedAt":"2026-08-14T22:57:27.446Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}