D4Vinci/Scrapling · error · RuntimeError

Cannot compare requests before generating their fingerprints

Error message

Cannot compare requests before generating their fingerprints!

What it means

Request.__eq__ compares requests by fingerprint, but the fingerprint (_fp) is computed lazily. If two Request objects are compared with == (or used in a set / as dict keys, which invokes __hash__ via the same state) before the fingerprint was ever generated, the comparison is impossible and raises RuntimeError.

Source

Thrown at scrapling/spiders/request.py:150

    def __lt__(self, other: object) -> bool:
        """Compare requests by priority"""
        if not isinstance(other, Request):
            return NotImplemented
        return self.priority < other.priority

    def __gt__(self, other: object) -> bool:
        """Compare requests by priority"""
        if not isinstance(other, Request):
            return NotImplemented
        return self.priority > other.priority

    def __eq__(self, other: object) -> bool:
        """Requests are equal if they have the same fingerprint."""
        if not isinstance(other, Request):
            return NotImplemented
        if self._fp is None or other._fp is None:
            raise RuntimeError("Cannot compare requests before generating their fingerprints!")
        return self._fp == other._fp

    def __getstate__(self) -> dict[str, Any]:
        """Prepare state for pickling - store callback as name string for pickle compatibility."""
        state = self.__dict__.copy()
        state["_callback_name"] = getattr(self.callback, "__name__", None) if self.callback is not None else None
        state["callback"] = None  # Don't pickle the actual callable
        return state

    def __setstate__(self, state: dict[str, Any]) -> None:
        """Restore state from pickle - callback restored later via _restore_callback()."""
        self._callback_name: str | None = state.pop("_callback_name", None)
        self.__dict__.update(state)

    def _restore_callback(self, spider: "Spider") -> None:
        """Restore callback from spider after unpickling.

        :param spider: Spider instance to look up callback method on

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Generate the fingerprint explicitly before comparing: _ = req_a.fingerprint; _ = req_b.fingerprint (or whatever the documented accessor/trigger is — accessing req.fingerprint populates _fp).
2. In dedup code, key on req.fingerprint directly instead of relying on __eq__/sets of Request objects.
3. If you control the flow, funnel requests through the framework's scheduling step first, which fingerprints them.

Example fix

# before
if req_a == req_b:  # RuntimeError if fingerprints not generated

# after
if req_a.fingerprint == req_b.fingerprint:  # accessors compute _fp lazily

# force lazy fingerprint generation before any comparison or set/dict use
_ = req_a.fingerprint
_ = req_b.fingerprint

seen = {req.fingerprint for req in requests}  # key on fingerprint, not Request

def has_fingerprint(req) -> bool:
    return getattr(req, '_fp', None) is not None

try:
    same = req_a == req_b
except RuntimeError:
    _ = req_a.fingerprint, req_b.fingerprint
    same = req_a == req_b

When it happens

Trigger: Creating two Request objects and immediately checking req_a == req_b, or putting fresh Requests into a set(), without touching req.fingerprint (or a scheduler step) first.

Common situations: Writing unit tests for dedup logic with hand-built Request objects; grouping requests manually before handing them to the spider scheduler that would normally generate fingerprints.

Related errors

• Can't convert {type(value).__name__} to bytes

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/ad99e87d09c71e57. Report an issue: GitHub.