unclecode/crawl4ai · error · RuntimeError

Failed to download PDF from {url}: {str(e)}

Error message

Failed to download PDF from {url}: {str(e)}

What it means

Catch-all RuntimeError raised when any non-timeout exception occurs while the PDF crawler strategy downloads a remote PDF via requests. The temp file is cleaned up (unlink + removal from _temp_files) before raising, so disk state stays consistent. The message wraps the original exception text, which is the key to diagnosis.

Source

Thrown at crawl4ai/processors/pdf/__init__.py:188

                            progress = (downloaded / total_size) * 100
                            if progress % 10 < 0.1:  # Log every 10%
                                self.logger.debug(f"PDF download progress: {progress:.0f}%")
                
                if self.logger:
                    self.logger.info(f"PDF downloaded successfully: {temp_file.name}")
                        
                return temp_file.name
                
            except requests.exceptions.Timeout as e:
                # Clean up temp file if download fails
                Path(temp_file.name).unlink(missing_ok=True)
                self._temp_files.remove(temp_file.name)
                raise RuntimeError(f"Timeout downloading PDF from {url}: {str(e)}")
            except Exception as e:
                # Clean up temp file if download fails
                Path(temp_file.name).unlink(missing_ok=True)
                self._temp_files.remove(temp_file.name)
                raise RuntimeError(f"Failed to download PDF from {url}: {str(e)}")
                
        elif url.startswith("file://"):
            return url[7:]  # Strip file:// prefix
            
        return url  # Assume local path
    

__all__ = ["PDFCrawlerStrategy", "PDFContentScrapingStrategy"]

View on GitHub (pinned to 7e80152142)

Solutions

  1. Read the wrapped {str(e)} portion of the message — it identifies the root cause (DNS, TLS, HTTP status, disk).
  2. Test the URL directly: curl -L -o /dev/null -w '%{http_code}' <url> to confirm reachability and content type.
  3. For TLS errors, update certifi/CA bundle or disable verification only if the environment requires it.
  4. For disk errors, free space or point TMPDIR at a volume with capacity; large PDFs can exceed small temp partitions.
  5. If the URL requires auth or specific headers, fetch it yourself and pass a local path or file:// URL (the strategy passes those through untouched).

Example fix

# before
result = await crawler.arun(url="https://example.com/reports/report.pdf")  # RuntimeError: Failed to download PDF ... SSLError

# after
import requests, pathlib
resp = requests.get(url, headers={"Authorization": "Bearer ..."}, timeout=120)
resp.raise_for_status()
local = pathlib.Path("/tmp/report.pdf"); local.write_bytes(resp.content)
result = await crawler.arun(url=local.as_uri())  # file:// path bypasses the downloader
Defensive patterns

Strategy: try-catch

Validate before calling

import requests

def check_pdf_download(url: str) -> tuple[bool, str]:
    try:
        r = requests.get(url, timeout=30, stream=True)
        r.raise_for_status()
        ct = r.headers.get("content-type", "")
        return ("pdf" in ct.lower(), ct)
    except requests.RequestException as e:
        return (False, str(e))

Try / catch

try:
    result = await crawler.arun(url=pdf_url)
except RuntimeError as e:
    if "Failed to download PDF" in str(e):
        logger.error(f"PDF fetch failed: {str(e).split(': ', 1)[-1]}")  # root cause is in the wrapped text
        mark_url_failed(pdf_url)

Prevention

When it happens

Trigger: Passing an http(s) URL to the PDF download step that triggers requests exceptions other than Timeout: ConnectionError (DNS failure, refused connection), HTTPError-class failures surfaced during streaming, TLS/SSL verification errors, chunked decode errors, or disk errors writing the temp file.

Common situations: 404/dead links, mistyped domains (DNS failure), corporate proxies or TLS interception breaking HTTPS, missing CA certificates, disk full while writing the temp file, or remote servers dropping the connection mid-download.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/33382a3cb3e12e31. Report an issue: GitHub.