{"record":{"id":"a257bdd56fe6bbdd","repo":"scrapy/scrapy","slug":"incorrect-uri-scheme-in-uri-expected-s3","errorCode":null,"errorMessage":"Incorrect URI scheme in {uri}, expected 's3'","messagePattern":"Incorrect URI scheme in (.+?), expected 's3'","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scrapy/pipelines/files.py","lineNumber":198,"sourceCode":"        config = (\n            Config(max_pool_connections=self.AWS_MAX_POOL_CONNECTIONS)\n            if self.AWS_MAX_POOL_CONNECTIONS is not None\n            else None\n        )\n        session = botocore.session.get_session()\n        self.s3_client = session.create_client(\n            \"s3\",\n            aws_access_key_id=self.AWS_ACCESS_KEY_ID,\n            aws_secret_access_key=self.AWS_SECRET_ACCESS_KEY,\n            aws_session_token=self.AWS_SESSION_TOKEN,\n            endpoint_url=self.AWS_ENDPOINT_URL,\n            region_name=self.AWS_REGION_NAME,\n            use_ssl=self.AWS_USE_SSL,\n            verify=self.AWS_VERIFY,\n            config=config,\n        )\n        if not uri.startswith(\"s3://\"):\n            raise ValueError(f\"Incorrect URI scheme in {uri}, expected 's3'\")\n        self.bucket, self.prefix = uri[5:].split(\"/\", 1)\n\n    @staticmethod\n    def _onsuccess(boto_key: dict[str, Any]) -> StatInfo:\n        checksum = boto_key[\"ETag\"].strip('\"')\n        last_modified = boto_key[\"LastModified\"]\n        modified_stamp = time.mktime(last_modified.timetuple())\n        return {\"checksum\": checksum, \"last_modified\": modified_stamp}\n\n    def stat_file(\n        self, path: str, info: MediaPipeline.SpiderInfo\n    ) -> Deferred[StatInfo]:\n\n        return self._get_boto_key(path).addCallback(self._onsuccess)\n\n    def _get_boto_key(self, path: str) -> Deferred[dict[str, Any]]:\n        key_name = f\"{self.prefix}{path}\"\n        return deferred_from_coro(","sourceCodeStart":180,"sourceCodeEnd":216,"githubUrl":"https://github.com/scrapy/scrapy/blob/06af687662112027b4482d31e2714a3cf280a91f/scrapy/pipelines/files.py#L180-L216","documentation":"S3FilesStore.__init__ raises ValueError when the store URI does not start with 's3://'. The class is selected from the URI scheme, so reaching this error means the scheme check and the URI content disagree — typically a misrouted store class or a malformed FILES_STORE. After the check, the code splits uri[5:] into bucket and prefix, so the scheme is load-bearing.","triggerScenarios":"Explicitly constructing S3FilesStore('gs://bucket') or S3FilesStore('/local/path'); a FILES_STORE like 's3:/bucket' (single slash) or with whitespace/newline from a .env file.","commonSituations":"Copy-paste of store URIs with typos; trailing newline in environment variables; subclassing FilesPipeline with a hardcoded store class while the setting points at another scheme.","solutions":["Set FILES_STORE to a well-formed S3 URI: 's3://bucket-name/prefix'.","Strip whitespace/newlines from environment-sourced settings: FILES_STORE.strip().","Don't hardcode S3FilesStore for non-S3 URIs; let FilesPipeline._get_store pick the class from the scheme."],"exampleFix":"# before\nFILES_STORE = os.environ['FILES_STORE']  # 's3://bucket\\n' -> ValueError\n\n# after\nFILES_STORE = os.environ['FILES_STORE'].strip()  # 's3://bucket'","handlingStrategy":"validation","validationCode":"store = FILES_STORE.strip() if isinstance(FILES_STORE, str) else ''\nif store.startswith('s3://'):\n    bucket, _, prefix = store[5:].partition('/')\n    assert bucket, 's3 URI must include a bucket: s3://bucket/prefix'","typeGuard":"def is_s3_uri(uri: str) -> bool:\n    return isinstance(uri, str) and uri.startswith('s3://') and len(uri[5:].split('/', 1)[0]) > 0","tryCatchPattern":null,"preventionTips":["Centralize store URI construction in one helper that validates the scheme.","Strip env-var-sourced URIs to kill trailing newlines.","Let FilesPipeline pick the store class from the URI scheme instead of hardcoding."],"tags":["scrapy","s3","configuration","uri","files-pipeline"],"backgroundTag":null,"analyzedSha":"06af687662112027b4482d31e2714a3cf280a91f","analyzedAt":"2026-08-15T00:21:48.041Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}