crewAIInc/crewAI · error · Exception

HTTP {response.status}: {response.reason}

Error message

HTTP {response.status}: {response.reason}

What it means

ArxivPaperTool.fetch_arxiv_data() opens the arXiv API URL with urllib and checks response.status; any non-200 status raises a generic Exception with the code and reason. In practice urllib raises HTTPError itself for most non-2xx statuses, so this branch fires mainly with non-standard status handling — it is a defensive check on the response object after urlopen succeeds.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/arxiv_paper_tool/arxiv_paper_tool.py:89

            results = [self._format_paper_result(p) for p in papers]
            return "\n\n" + "-" * 80 + "\n\n".join(results)

        except Exception as e:
            logger.error(f"ArxivTool Error: {e!s}")
            return f"Failed to fetch or download Arxiv papers: {e!s}"

    def fetch_arxiv_data(
        self, search_query: str, max_results: int
    ) -> list[dict[str, Any]]:
        api_url = f"{self.BASE_API_URL}?search_query={urllib.parse.quote(search_query)}&start=0&max_results={max_results}"
        logger.info(f"Fetching data from Arxiv API: {api_url}")

        try:
            with urllib.request.urlopen(  # noqa: S310
                api_url, timeout=self.REQUEST_TIMEOUT
            ) as response:
                if response.status != 200:
                    raise Exception(f"HTTP {response.status}: {response.reason}")
                data = response.read().decode("utf-8")
        except urllib.error.URLError as e:
            logger.error(f"Error fetching data from Arxiv: {e}")
            raise

        root = ET.fromstring(data)  # noqa: S314
        papers = []

        for entry in root.findall(self.ATOM_NAMESPACE + "entry"):
            raw_id = self._get_element_text(entry, "id")
            arxiv_id = raw_id.split("/")[-1].replace(".", "_") if raw_id else "unknown"

            title = self._get_element_text(entry, "title") or "No Title"
            summary = self._get_element_text(entry, "summary") or "No Summary"
            published = self._get_element_text(entry, "published") or "No Publish Date"
            authors = [
                self._get_element_text(author, "name") or "Unknown"
                for author in entry.findall(self.ATOM_NAMESPACE + "author")

View on GitHub (pinned to 754d7323be)

Solutions

1. Retry after a delay — arXiv intermittently returns 503 under load; back off (e.g. 5-15s) and reduce request frequency.
2. Verify the API URL by printing/logging it and opening it in a browser or with curl to see the raw status.
3. Respect arXiv's API etiquette: add ~3s between requests and cache results.
4. If persistent, check https://status.arxiv.org for outages.

Example fix

# before
tool = ArxivPaperTool()
papers = tool.fetch_arxiv_data("llm agents", 10)

# after
import time
for attempt in range(3):
    try:
        papers = tool.fetch_arxiv_data("llm agents", 10)
        break
    except Exception as e:
        if attempt == 2:
            raise
        time.sleep(10)  # arXiv rate limits with 503s

try:
    papers = tool.fetch_arxiv_data(query, 10)
except Exception as e:
    if "HTTP 503" in str(e) or "HTTP 429" in str(e):
        time.sleep(15)
        papers = tool.fetch_arxiv_data(query, 10)
    else:
        raise

When it happens

Trigger: arXiv API returning an error status (e.g. 503 during rate limiting / maintenance windows, 400 for a malformed query string that still built a valid URL); servers or test doubles returning non-200 without urllib's default HTTPError behavior.

Common situations: Hammering the arXiv API without throttling and getting throttled; arXiv downtime; a search_query string with characters that survive urllib.parse.quote but still produce a bad request.

Related errors

• Brave Search API error (HTTP {status}): {body}
• Project name '{name}' would generate invalid Python class na
• Failed to retrieve organization list: {e!s}
• Unable to fetch documentation from {docs_url}: {e}
• Error fetching content from URL {url}: {e!s}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/ec75044f1cbccd0c. Report an issue: GitHub.