crewAIInc/crewAI · error · ValueError

URL directory loading is not supported. Please provide a loc

Error message

URL directory loading is not supported. Please provide a local directory path.

What it means

DirectoryLoader.load() explicitly refuses URL inputs: if source_content.is_url() is true it raises ValueError telling the user to supply a local directory path. Directory listing over HTTP is not implemented, so pointing the DIRECTORY data type at a web URL fails immediately — before the os.path.exists check.

Source

Thrown at lib/crewai-tools/src/crewai_tools/rag/loaders/directory_loader.py:24

from crewai_tools.rag.source_content import SourceContent


class DirectoryLoader(BaseLoader):
    def load(self, source_content: SourceContent, **kwargs: Any) -> LoaderResult:  # type: ignore[override]
        """Load and process all files from a directory recursively.

        Args:
            source_content: Directory path or URL to a directory listing
            **kwargs: Additional options:
                - recursive: bool (default True) - Whether to search recursively
                - include_extensions: list - Only include files with these extensions
                - exclude_extensions: list - Exclude files with these extensions
                - max_files: int - Maximum number of files to process
        """
        source_ref = source_content.source_ref

        if source_content.is_url():
            raise ValueError(
                "URL directory loading is not supported. Please provide a local directory path."
            )

        if not os.path.exists(source_ref):
            raise FileNotFoundError(f"Directory does not exist: {source_ref}")

        if not os.path.isdir(source_ref):
            raise ValueError(f"Path is not a directory: {source_ref}")

        return self._process_directory(source_ref, kwargs)

    def _process_directory(self, dir_path: str, kwargs: dict[str, Any]) -> LoaderResult:
        recursive: bool = kwargs.get("recursive", True)
        include_extensions: list[str] | None = kwargs.get("include_extensions", None)
        exclude_extensions: list[str] | None = kwargs.get("exclude_extensions", None)
        max_files: int | None = kwargs.get("max_files", None)

        files = self._find_files(

View on GitHub (pinned to 754d7323be)

Solutions

1. Download/sync the remote directory locally first (aws s3 sync, wget -r, git clone) and pass the local path
2. If the target is a docs website, use DataType.DOCS_SITE instead; for a single page use WEBSITE
3. Mount or clone the remote tree in CI before running ingestion

Example fix

# before
result = DirectoryLoader().load(SourceContent('https://example.com/docs/'))
# after (shell): git clone https://github.com/org/docs ./docs
# code:
result = DirectoryLoader().load(SourceContent('./docs'))

from urllib.parse import urlparse
if urlparse(source).scheme in ('http', 'https'):
    raise ValueError('directory source must be a local path; clone/sync the remote tree first')

def is_local_directory(p: str) -> bool:
    from pathlib import Path
    u = urlparse(p)
    return not u.scheme and Path(p).is_dir()

try:
    result = DirectoryLoader().load(source_content)
except ValueError as e:
    if 'URL directory loading' in str(e):
        # fall back to docs-site crawling for web content
        loader = DataType.DOCS_SITE.get_loader()
    else:
        raise

When it happens

Trigger: DataTypes.from_content classifying input as DataType.DIRECTORY while the source is an http(s) URL (e.g. 'https://example.com/files/' passed with a directory-like path), or explicitly constructing a directory source from a URL and calling load().

Common situations: Mirroring an S3/HTTP bucket layout and passing its URL expecting recursive fetch; config reuse where a local path was replaced by a hosted one; URL that ends with a slash misclassified as a directory.

Related errors

• Directory does not exist: {source_ref}
• Directory does not exist: {source_ref}
• Path is not a directory: {source_ref}
• Invalid GitHub URL: {repo_url}
• Invalid GitHub repository URL: {repo_url}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/5e51b80793c630ed. Report an issue: GitHub.