datawhalechina/hello-agents · error · ExternalAPIException

ArXiv API请求失败: {response.status}

Error message

ArXiv API请求失败: {response.status}

What it means

ExternalAPIException raised in HunterAgent._search_papers_from_arxiv when the GET to arxiv_base_url (export.arxiv.org/api/query) returns any status other than 200, with the status code in the message. arXiv's public API has no key; non-200 almost always means rate limiting (429), temporary unavailability (503), or a malformed/oversized query string.

Source

Thrown at Co-creation-projects/Apricity-InnocoreAI/agents/hunter.py:122

        # 添加时间过滤
        date_filter = ""
        if days_back > 0:
            start_date = (datetime.now() - timedelta(days=days_back)).strftime("%Y%m%d")
            date_filter = f"submittedDate:[{start_filter}0000 TO {datetime.now().strftime('%Y%m%d')}2359]"
        
        params = {
            "search_query": query,
            "start": 0,
            "max_results": max_papers * 2,  # 获取更多结果以便筛选
            "sortBy": "submittedDate",
            "sortOrder": "descending"
        }
        
        try:
            async with aiohttp.ClientSession() as session:
                async with session.get(self.arxiv_base_url, params=params) as response:
                    if response.status != 200:
                        raise ExternalAPIException(f"ArXiv API请求失败: {response.status}")
                    
                    xml_content = await response.text()
                    feed = feedparser.parse(xml_content)
                    
                    for entry in feed.entries:
                        paper = {
                            "id": entry.id.split("/")[-1],
                            "title": entry.title,
                            "authors": [author.name for author in entry.authors],
                            "abstract": entry.summary,
                            "published": entry.published,
                            "pdf_url": entry.link.replace('/abs/', '/pdf/') + '.pdf',
                            "source": "arxiv",
                            "doi": entry.get('arxiv_doi', ''),
                            "categories": [tag.term for tag in entry.tags]
                        }
                        
                        papers.append(paper)

View on GitHub (pinned to 606a07d341)

Solutions

  1. On 429/503: back off (sleep 3-10s, exponential) and retry the request a few times before surfacing the error.
  2. Simplify/encode the search_query: quote keywords, limit boolean nesting, keep max_results ≤ 100 per call and page with 'start'.
  3. Cache arXiv responses (same query within N minutes) so repeated runs don't re-hit the API.
  4. Run from a stable IP or route through a polite cache (e.g. arXiv-compatible mirrors) when doing bulk harvests.
  5. Check https://status.arxiv.org for outages when 5xx persists.

Example fix

# before
async with session.get(self.arxiv_base_url, params=params) as response:
    if response.status != 200:
        raise ExternalAPIException(f"ArXiv API请求失败: {response.status}")

# after — honor Retry-After / back off on throttling
import asyncio
for attempt in range(4):
    async with session.get(self.arxiv_base_url, params=params) as response:
        if response.status == 200:
            xml_content = await response.text()
            break
        if response.status in (429, 500, 503) and attempt < 3:
            retry_after = int(response.headers.get("Retry-After", 3 * (attempt + 1)))
            await asyncio.sleep(retry_after)
            continue
        raise ExternalAPIException(f"ArXiv API请求失败: {response.status}")
Defensive patterns

Strategy: retry

Validate before calling

# Sanity-check the query before it leaves
from urllib.parse import quote
assert len(keywords) > 0, "keywords required"
query = " AND ".join(f"all:{quote(k)}" for k in keywords)
assert len(query) < 300, "arXiv search_query too long; split into batches"

Try / catch

from agents.exceptions import ExternalAPIException
try:
    papers = await hunter._search_papers_from_arxiv(kws, max_papers, days_back)
except ExternalAPIException as e:
    if any(code in str(e) for code in ("429", "500", "503")):
        await asyncio.sleep(backoff())
        papers = await hunter._search_papers_from_arxiv(kws, max_papers, days_back)
    else:
        raise

Prevention

When it happens

Trigger: Burst searches (max_results doubled by the code to max_papers*2) from one IP exceeding arXiv's ~1 req/3s courtesy limit -> 429; long multi-keyword query string with unencoded operators returning 400; arXiv maintenance window returning 5xx; corporate proxy intercepting with 403.

Common situations: Batch jobs searching dozens of keyword sets back-to-back; special characters (quotes, AND/OR braces) in keywords not URL-encoded by params; running hunter from cloud IPs that arXiv throttles harder.

Related errors

AI-assisted analysis of datawhalechina/hello-agents@606a07d341 (2026-08-14). Data as JSON: /api/errors/e1f4b443070e8d2c. Report an issue: GitHub.