D4Vinci/Scrapling · error · TypeError

Can't pickle Selector objects

Error message

Can't pickle Selector objects

What it means

Raised by Selector.__getstate__ whenever something attempts to pickle a Selector (pickle.dump, copy.deepcopy, multiprocessing queues, joblib, cloudpickle). The underlying lxml HtmlElement tree cannot be serialized, so the class deliberately fails fast with a TypeError instead of producing a corrupt pickle.

Source

Thrown at scrapling/parser.py:252

                adaptive=adaptive,
                _storage=storage,
                keep_comments=comments,
                keep_cdata=cdata,
                huge_tree=huge_tree,
            )
            for el in elements
        )

    def __handle_elements(self, result: List[HtmlElement | _ElementUnicodeResult]) -> "Selectors":
        """Used internally in all functions to convert results to Selectors in bulk"""
        if not result:
            return Selectors()

        return self.__elements_convertor(result)

    def __getstate__(self) -> Any:
        # lxml don't like it :)
        raise TypeError("Can't pickle Selector objects")

    # The following four properties I made them into functions instead of variables directly
    # So they don't slow down the process of initializing many instances of the class and gets executed only
    # when the user needs them for the first time for that specific element and gets cached for next times
    # Doing that only made the library performance test sky rocked multiple times faster than before
    # because I was executing them on initialization before :))
    @property
    def tag(self) -> str:
        """Get the tag name of the element"""
        if self._is_text_node(self._root):
            return "#text"
        if not self.__tag:
            self.__tag = str(self._root.tag)
        return self.__tag or ""

    @property
    def text(self) -> TextHandler:
        """Get text content of the element"""

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Extract plain data (strings, dicts) from the Selector before pickling, e.g. store element.html, element.attrib, or text content instead of the Selector itself.
2. If multiprocessing is only used for CPU parallelism, switch to threading/asyncio since parsing is C-bound (lxml releases the GIL) and objects never need to cross process boundaries.
3. As a last resort for process transfer, transfer the raw HTML bytes and re-parse with Selector(content) in the worker.

Example fix

// before
import pickle
pickle.dumps(page.css_first('h1'))  # TypeError

// after
import pickle
data = page.css_first('h1').html  # str is picklable
pickle.dumps(data)

def picklable_extraction(sel):
    # reduce to primitives before any pickle/deepcopy boundary
    return {
        'tag': sel.tag,
        'text': sel.text,
        'attrib': dict(sel.attrib),
        'html': sel.html,
    }

results = [picklable_extraction(s) for s in page.css('a')]
pickle.dumps(results)  # safe

from scrapling.parser import Selector

def is_selector(obj) -> bool:
    # true means: do NOT pickle obj directly; extract data first
    return isinstance(obj, Selector)

try:
    pickle.dumps(obj)
except TypeError as e:
    if "Can't pickle Selector" in str(e):
        raise ValueError('Extract primitive data from Selector before pickling') from e
    raise

When it happens

Trigger: Calling pickle.dumps(selector), copy.deepcopy(selector), passing Selector objects between multiprocessing.Process workers, or using libraries like joblib/pandas that pickle their inputs.

Common situations: Scaling a scraper with multiprocessing and sending Selector results through a Pool; caching parsed pages with pickle; deepcopy of a dict that contains Selector objects.

Related errors

• Can't pickle Selectors object
• Invalid XPath selector: {selector}

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/4fe6e7631aa6b3b8. Report an issue: GitHub.