D4Vinci/Scrapling · error · TypeError

Response meta should be dictionary but got {type(meta).__nam

Error message

Response meta should be dictionary but got {type(meta).__name__} instead!

What it means

Raised in the scrapling Response constructor (custom.py) when the meta argument is truthy but not a dict. Response metadata is stored as self.meta and consumed as a mapping by the crawler and follow() logic, so a non-dict truthy value (list, string, object) is rejected with a TypeError naming the actual type.

Source

Thrown at scrapling/engines/toolbelt/custom.py:77

        adaptive_domain: str = cast(str, selector_config.pop("adaptive_domain", ""))
        self.status = status
        self.reason = reason
        self.cookies = cookies
        self.headers = headers
        self.request_headers = request_headers
        self.history = history or []
        super().__init__(
            content=content,
            url=adaptive_domain or url,
            encoding=encoding,
            **selector_config,
        )
        # For easier debugging while working from a Python shell
        log.info(f"Fetched ({status}) <{method} {url}> (referer: {request_headers.get('referer')})")

        if meta and not isinstance(meta, dict):
            raise TypeError(f"Response meta should be dictionary but got {type(meta).__name__} instead!")

        self.meta: Dict[str, Any] = meta or {}
        self.request: Optional["Request"] = None  # Will be set by crawler
        self.captured_xhr: List["Response"] = []

    @property
    def body(self) -> bytes:
        """Return the raw body of the response as bytes."""
        return cast(bytes, cast(Sequence, self._raw_body))

    def follow(
        self,
        url: str,
        sid: str = "",
        callback: Callable[["Response"], AsyncGenerator[Union[Dict[str, Any], "Request", None], None]] | None = None,
        priority: int | None = None,
        dont_filter: bool = False,
        meta: dict[str, Any] | None = None,

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Pass meta as a plain dict: meta={'key': 'value'}.
2. If the value arrives serialized, parse it first: meta=json.loads(raw).
3. Omit meta entirely when you have nothing to store — it defaults to {}.

Example fix

# before
resp = Response(status, url, content, meta=json.dumps({'proxy': p}))  # TypeError: str

# after
resp = Response(status, url, content, meta={'proxy': p})

if meta is not None and not isinstance(meta, dict):
    raise TypeError(f'meta must be dict, got {type(meta).__name__}')
resp = Response(status, url, content, meta=meta)

def is_valid_meta(meta) -> bool:
    return meta is None or isinstance(meta, dict)

try:
    resp = Response(status, url, content, meta=meta)
except TypeError as e:
    if 'meta should be dictionary' in str(e):
        resp = Response(status, url, content)  # proceed without meta
    else:
        raise

When it happens

Trigger: Passing Response(..., meta=['key', 'value']) or meta='data' or meta=SomeObj(); spider callbacks constructing Response objects manually with malformed meta; passing a JSON string instead of a parsed dict.

Common situations: Hand-building Response objects in custom fetchers/tests; copying examples where meta was serialized (json.dumps) and forgetting to parse it back; passing a dataclass where a dict was expected.

Related errors

• This response has no request set yet.
• Invalid proxy string: {proxy_string}
• strategy must be callable, got {type(strategy).__name__}
• Invalid proxy type: {type(proxy)}. Expected str or dict.
• Argument `selector_config` must be a dictionary.

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/ec3ee181aa59df5b. Report an issue: GitHub.