D4Vinci/Scrapling · error · TypeError

Can't pickle Selectors object

Error message

Can't pickle Selectors object

What it means

Selectors (the list-like result container returned by css()/xpath()/find_all()) implements __getstate__ to raise TypeError on any pickling attempt. Each item wraps an lxml HtmlElement, which cannot be serialized, so copying or pickling the whole collection is blocked the same way as for a single Selector.

Source

Thrown at scrapling/parser.py:1376

    @property
    def first(self) -> Optional[Selector]:
        """Returns the first Selector item of the current list or `None` if the list is empty"""
        return self[0] if len(self) > 0 else None

    @property
    def last(self) -> Optional[Selector]:
        """Returns the last Selector item of the current list or `None` if the list is empty"""
        return self[-1] if len(self) > 0 else None

    @property
    def length(self) -> int:
        """Returns the length of the current list"""
        return len(self)

    def __getstate__(self) -> Any:  # pragma: no cover
        # lxml don't like it :)
        raise TypeError("Can't pickle Selectors object")


# For backward compatibility
Adaptor = Selector
Adaptors = Selectors

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Convert to plain data before crossing a serialization boundary: [{'tag': s.tag, 'text': s.text, 'attrs': s.attrib} for s in results].
2. Use .html strings to carry element fragments across processes and re-parse on the other side.
3. Keep parsing inside one process; ship only URLs/HTML in, extracted primitives out.

Example fix

# before
results = page.css('a')
pickle.dumps(results)  # TypeError

# after
results = [{'text': s.text, 'href': s.attrib.get('href')} for s in page.css('a')]
pickle.dumps(results)

results = page.css('a')
picklable = [s.html for s in results]          # str fragments
# or full data:
picklable = [{'tag': s.tag, 'text': s.text, 'attrib': dict(s.attrib)} for s in results]

from scrapling.parser import Selectors

def is_selectors(obj) -> bool:
    # true means: convert to primitives before pickling
    return isinstance(obj, (Selectors, list)) and any(type(x).__name__ == 'Selector' for x in (obj if isinstance(obj, list) else list(obj)))

try:
    pickle.dumps(payload)
except TypeError as e:
    if "Can't pickle Selectors" in str(e):
        payload = [s.html for s in payload]
    else:
        raise

When it happens

Trigger: pickle.dumps(page.css('a')), copy.deepcopy(results), multiprocessing.Pool.map over functions returning Selectors, or asyncio task result pickling.

Common situations: Parallelizing scraping across processes and returning matched element lists; memoizing query results; joblib caching of extraction functions.

Related errors

• Can't pickle Selector objects

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/d3d8e51f908f2a64. Report an issue: GitHub.