scrapy/scrapy · error · TypeError

{self.files_urls_field} must be a list of URLs, got {type(ur

Error message

{self.files_urls_field} must be a list of URLs, got {type(urls).__name__}.

What it means

FilesPipeline.get_media_requests raises TypeError when the item's file_urls field (files_urls_field, default 'file_urls') is not a list. The pipeline iterates the field to build Requests, and since the field type is part of the contract, anything else (str, dict, tuple, None-with-wrong-adapter) is rejected. Each element is then passed to Request(u, ...), so element types matter too.

Source

Thrown at scrapy/pipelines/files.py:723

        request: Request,
        info: MediaPipeline.SpiderInfo,
        *,
        item: Any = None,
    ) -> str:
        path = self.file_path(request, response=response, info=info, item=item)
        buf = BytesIO(response.body)
        checksum = _md5sum(buf)
        buf.seek(0)
        await ensure_awaitable(self.store.persist_file(path, buf, info))
        return checksum

    # Overridable Interface
    def get_media_requests(
        self, item: Any, info: MediaPipeline.SpiderInfo
    ) -> list[Request]:
        urls = ItemAdapter(item).get(self.files_urls_field, [])
        if not isinstance(urls, list):
            raise TypeError(
                f"{self.files_urls_field} must be a list of URLs, got {type(urls).__name__}. "
            )
        return [Request(u, callback=NO_CALLBACK) for u in urls]

    def file_downloaded(
        self,
        response: Response,
        request: Request,
        info: MediaPipeline.SpiderInfo,
        *,
        item: Any = None,
    ) -> str | Awaitable[str]:
        return self._file_downloaded(response, request, info, item=item)

    def item_completed(
        self, results: list[FileInfoOrError], item: Any, info: MediaPipeline.SpiderInfo
    ) -> Any:
        with suppress(KeyError):

View on GitHub (pinned to 06af687662)

Solutions

Make the field a list of URL strings: item['file_urls'] = [url1, url2].
Coerce in a pre-pipeline or in your spider: item['file_urls'] = list(urls).
If you renamed the field with FILES_URLS_FIELD, ensure items populate the new field name with a list.
Wrap a single URL in a list.

Example fix

# before
item['file_urls'] = 'http://example.com/doc.pdf'  # str -> TypeError

# after
item['file_urls'] = ['http://example.com/doc.pdf']

Defensive patterns

Strategy: type-guard

Validate before calling

urls = ItemAdapter(item).get(pipeline.files_urls_field, [])
assert isinstance(urls, list), f'expected list, got {type(urls).__name__}'

Type guard

def is_url_list(value) -> bool:
    return isinstance(value, list) and all(isinstance(u, str) for u in value)

Try / catch

from scrapy.pipelines.media import MediaPipeline
# TypeError here is not caught by the pipeline; fix the item upstream instead:
if not isinstance(item.get('file_urls', []), list):
    item['file_urls'] = [item['file_urls']] if isinstance(item['file_urls'], str) else list(item['file_urls'])

Prevention

Normalize file_urls to a list in the spider or a pre-pipeline.
Define item fields with type hints/docs that say list[str].
Add a quick unit test asserting your yielded items pass the list check.

When it happens

Trigger: Yielding an item where file_urls is a single URL string, a tuple, or a dict; renaming the field via FILES_URLS_FIELD but populating the old one; custom items that lazily compute a generator instead of a list.

Common situations: Beginner items setting file_urls = 'http://site/f.pdf'; pipelines that transform URLs into tuples; dataclasses defaulting the field to a string; item adapters returning non-list containers.

Related errors

AI-assisted analysis of scrapy/scrapy@06af687662 (2026-08-15). Data as JSON: /api/errors/9839d3af04fa80bf. Report an issue: GitHub.