{"record":{"id":"5e51b80793c630ed","repo":"crewAIInc/crewAI","slug":"url-directory-loading-is-not-supported-please-pro","errorCode":null,"errorMessage":"URL directory loading is not supported. Please provide a local directory path.","messagePattern":"URL directory loading is not supported\\. Please provide a local directory path\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/rag/loaders/directory_loader.py","lineNumber":24,"sourceCode":"from crewai_tools.rag.source_content import SourceContent\n\n\nclass DirectoryLoader(BaseLoader):\n    def load(self, source_content: SourceContent, **kwargs: Any) -> LoaderResult:  # type: ignore[override]\n        \"\"\"Load and process all files from a directory recursively.\n\n        Args:\n            source_content: Directory path or URL to a directory listing\n            **kwargs: Additional options:\n                - recursive: bool (default True) - Whether to search recursively\n                - include_extensions: list - Only include files with these extensions\n                - exclude_extensions: list - Exclude files with these extensions\n                - max_files: int - Maximum number of files to process\n        \"\"\"\n        source_ref = source_content.source_ref\n\n        if source_content.is_url():\n            raise ValueError(\n                \"URL directory loading is not supported. Please provide a local directory path.\"\n            )\n\n        if not os.path.exists(source_ref):\n            raise FileNotFoundError(f\"Directory does not exist: {source_ref}\")\n\n        if not os.path.isdir(source_ref):\n            raise ValueError(f\"Path is not a directory: {source_ref}\")\n\n        return self._process_directory(source_ref, kwargs)\n\n    def _process_directory(self, dir_path: str, kwargs: dict[str, Any]) -> LoaderResult:\n        recursive: bool = kwargs.get(\"recursive\", True)\n        include_extensions: list[str] | None = kwargs.get(\"include_extensions\", None)\n        exclude_extensions: list[str] | None = kwargs.get(\"exclude_extensions\", None)\n        max_files: int | None = kwargs.get(\"max_files\", None)\n\n        files = self._find_files(","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/rag/loaders/directory_loader.py#L6-L42","documentation":"DirectoryLoader.load() explicitly refuses URL inputs: if source_content.is_url() is true it raises ValueError telling the user to supply a local directory path. Directory listing over HTTP is not implemented, so pointing the DIRECTORY data type at a web URL fails immediately — before the os.path.exists check.","triggerScenarios":"DataTypes.from_content classifying input as DataType.DIRECTORY while the source is an http(s) URL (e.g. 'https://example.com/files/' passed with a directory-like path), or explicitly constructing a directory source from a URL and calling load().","commonSituations":"Mirroring an S3/HTTP bucket layout and passing its URL expecting recursive fetch; config reuse where a local path was replaced by a hosted one; URL that ends with a slash misclassified as a directory.","solutions":["Download/sync the remote directory locally first (aws s3 sync, wget -r, git clone) and pass the local path","If the target is a docs website, use DataType.DOCS_SITE instead; for a single page use WEBSITE","Mount or clone the remote tree in CI before running ingestion"],"exampleFix":"# before\nresult = DirectoryLoader().load(SourceContent('https://example.com/docs/'))\n# after (shell): git clone https://github.com/org/docs ./docs\n# code:\nresult = DirectoryLoader().load(SourceContent('./docs'))","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\nif urlparse(source).scheme in ('http', 'https'):\n    raise ValueError('directory source must be a local path; clone/sync the remote tree first')","typeGuard":"def is_local_directory(p: str) -> bool:\n    from pathlib import Path\n    u = urlparse(p)\n    return not u.scheme and Path(p).is_dir()","tryCatchPattern":"try:\n    result = DirectoryLoader().load(source_content)\nexcept ValueError as e:\n    if 'URL directory loading' in str(e):\n        # fall back to docs-site crawling for web content\n        loader = DataType.DOCS_SITE.get_loader()\n    else:\n        raise","preventionTips":["Sync remote trees locally (git clone, aws s3 sync) before DIRECTORY ingestion","Choose DOCS_SITE/WEBSITE DataTypes for web-hosted content","Validate that the source is an existing local dir before starting long ingestion runs"],"tags":["rag","loader","directory","validation"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}