unclecode/crawl4ai · error · ValueError

Unsafe download filename rejected: {filename!r}

Error message

Unsafe download filename rejected: {filename!r}

What it means

Raised by _safe_download_filepath() when a suggested download filename escapes the downloads root after basename sanitization and realpath resolution. The function takes os.path.basename of the name, joins it to the realpath of downloads_path, and requires the commonpath of root and resolved path to equal the root; any residual escape (e.g. via a pre-existing symlink inside the directory pointing outside) raises ValueError.

Source

Thrown at crawl4ai/async_crawler_strategy.py:2438

def _safe_download_filepath(downloads_path: str, filename: str) -> str:
    """Resolve a download destination confined to ``downloads_path``.

    The filename is derived from attacker-influenced input (a remote
    Content-Disposition header, or the browser's suggested filename), so it is
    reduced to a bare basename (dropping absolute paths and ``..`` traversal)
    and the resolved path is re-checked to live inside the downloads root,
    rejecting any pre-existing symlink that points outside. Raises ValueError
    on any escape. The final write must still use ``_nofollow_opener`` (or an
    equivalent ``O_NOFOLLOW`` / pre-write symlink check) to close the TOCTOU
    window between this check and the open.
    """
    safe_name = os.path.basename(filename or "")
    if not safe_name or safe_name in (".", ".."):
        safe_name = f"download_{hashlib.md5((filename or '').encode()).hexdigest()[:10]}"
    real_root = os.path.realpath(downloads_path)
    real_path = os.path.realpath(os.path.join(real_root, safe_name))
    if os.path.commonpath([real_root, real_path]) != real_root:
        raise ValueError(f"Unsafe download filename rejected: {filename!r}")
    return real_path


def _nofollow_opener(path, flags):
    """Opener for ``open``/``aiofiles.open`` that refuses to follow a symlink at
    the final path component, closing the TOCTOU symlink-swap race after a path
    has been confined by ``_safe_download_filepath``."""
    return os.open(path, flags | os.O_NOFOLLOW)


class HTTPCrawlerError(Exception):
    """Base error class for HTTP crawler specific exceptions"""
    pass


class ConnectionTimeoutError(HTTPCrawlerError):
    """Raised when connection timeout occurs"""
    pass

View on GitHub (pinned to 7e80152142)

Solutions

1. Remove symlinks inside the downloads directory (find downloads_path -type l -delete) so realpath stays inside the root.
2. Point downloads_path at a real (non-symlinked) directory, avoiding macOS /tmp -> /private/tmp style mismatches between callers.
3. Use a per-run or per-process unique downloads directory.
4. Restrict write permissions on the downloads dir so only the crawler can create entries.

Example fix

// before
# downloads/data -> /etc/passwd  (symlink planted inside downloads dir)
# ValueError: Unsafe download filename rejected

// after
import os
for name in os.listdir(downloads_path):
    p = os.path.join(downloads_path, name)
    if os.path.islink(p):
        os.unlink(p)
await crawler.arun(url, config)

import os

def downloads_dir_clean(path: str) -> bool:
    rp = os.path.realpath(path)
    for root, _dirs, files in os.walk(rp):
        for f in files:
            if os.path.islink(os.path.join(root, f)):
                return False
    return True

try:
    await crawler.arun(url, config=cfg)
except ValueError as e:
    if "Unsafe download filename" in str(e):
        logger.error(f"possible symlink attack in downloads dir: {e}")

When it happens

Trigger: A pre-existing symlink named like the suggested filename inside the downloads directory that resolves outside the root (realpath follows it and commonpath check fails). Pure traversal via '../' in the filename is already neutralized by basename(), so the realistic trigger is the symlink-inside-directory case.

Common situations: Untrusted archives extracted into the downloads dir creating symlinks; multi-process crawlers sharing a downloads dir with differing roots (one root inside a symlinked path, e.g. /tmp vs /private/tmp on macOS); prior malicious downloads planting symlinks.

Related errors

• Refusing to write download through symlink: {download_path}
• Invalid wait_for parameter: '{wait_for}'. It should be eithe
• LLMConfig.api_token may not reference an environment variabl
• Rejected request: {e}
• invalid header name {name!r}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/abc43f1bd12999f2. Report an issue: GitHub.