scrapy/scrapy · error · ValueError

Incorrect URI scheme in {uri}, expected 's3'

Error message

Incorrect URI scheme in {uri}, expected 's3'

What it means

S3FilesStore.__init__ raises ValueError when the store URI does not start with 's3://'. The class is selected from the URI scheme, so reaching this error means the scheme check and the URI content disagree — typically a misrouted store class or a malformed FILES_STORE. After the check, the code splits uri[5:] into bucket and prefix, so the scheme is load-bearing.

Source

Thrown at scrapy/pipelines/files.py:198

        config = (
            Config(max_pool_connections=self.AWS_MAX_POOL_CONNECTIONS)
            if self.AWS_MAX_POOL_CONNECTIONS is not None
            else None
        )
        session = botocore.session.get_session()
        self.s3_client = session.create_client(
            "s3",
            aws_access_key_id=self.AWS_ACCESS_KEY_ID,
            aws_secret_access_key=self.AWS_SECRET_ACCESS_KEY,
            aws_session_token=self.AWS_SESSION_TOKEN,
            endpoint_url=self.AWS_ENDPOINT_URL,
            region_name=self.AWS_REGION_NAME,
            use_ssl=self.AWS_USE_SSL,
            verify=self.AWS_VERIFY,
            config=config,
        )
        if not uri.startswith("s3://"):
            raise ValueError(f"Incorrect URI scheme in {uri}, expected 's3'")
        self.bucket, self.prefix = uri[5:].split("/", 1)

    @staticmethod
    def _onsuccess(boto_key: dict[str, Any]) -> StatInfo:
        checksum = boto_key["ETag"].strip('"')
        last_modified = boto_key["LastModified"]
        modified_stamp = time.mktime(last_modified.timetuple())
        return {"checksum": checksum, "last_modified": modified_stamp}

    def stat_file(
        self, path: str, info: MediaPipeline.SpiderInfo
    ) -> Deferred[StatInfo]:

        return self._get_boto_key(path).addCallback(self._onsuccess)

    def _get_boto_key(self, path: str) -> Deferred[dict[str, Any]]:
        key_name = f"{self.prefix}{path}"
        return deferred_from_coro(

View on GitHub (pinned to 06af687662)

Solutions

1. Set FILES_STORE to a well-formed S3 URI: 's3://bucket-name/prefix'.
2. Strip whitespace/newlines from environment-sourced settings: FILES_STORE.strip().
3. Don't hardcode S3FilesStore for non-S3 URIs; let FilesPipeline._get_store pick the class from the scheme.

Example fix

# before
FILES_STORE = os.environ['FILES_STORE']  # 's3://bucket\n' -> ValueError

# after
FILES_STORE = os.environ['FILES_STORE'].strip()  # 's3://bucket'

store = FILES_STORE.strip() if isinstance(FILES_STORE, str) else ''
if store.startswith('s3://'):
    bucket, _, prefix = store[5:].partition('/')
    assert bucket, 's3 URI must include a bucket: s3://bucket/prefix'

def is_s3_uri(uri: str) -> bool:
    return isinstance(uri, str) and uri.startswith('s3://') and len(uri[5:].split('/', 1)[0]) > 0

When it happens

Trigger: Explicitly constructing S3FilesStore('gs://bucket') or S3FilesStore('/local/path'); a FILES_STORE like 's3:/bucket' (single slash) or with whitespace/newline from a .env file.

Common situations: Copy-paste of store URIs with typos; trailing newline in environment variables; subclassing FilesPipeline with a hardcoded store class while the setting points at another scheme.

Related errors

• Incorrect URI scheme in {uri}, expected 'ftp'
• missing botocore library
• Header "{key}" is not supported by botocore
• {setting_name} setting must be set to a valid path (not empt
• Invalid entry point {entry_point.name}

AI-assisted analysis of scrapy/scrapy@06af687662 (2026-08-15). Data as JSON: /api/errors/a257bdd56fe6bbdd. Report an issue: GitHub.