{"record":{"id":"4fe6e7631aa6b3b8","repo":"D4Vinci/Scrapling","slug":"can-t-pickle-selector-objects","errorCode":null,"errorMessage":"Can't pickle Selector objects","messagePattern":"Can't pickle Selector objects","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"scrapling/parser.py","lineNumber":252,"sourceCode":"                adaptive=adaptive,\n                _storage=storage,\n                keep_comments=comments,\n                keep_cdata=cdata,\n                huge_tree=huge_tree,\n            )\n            for el in elements\n        )\n\n    def __handle_elements(self, result: List[HtmlElement | _ElementUnicodeResult]) -> \"Selectors\":\n        \"\"\"Used internally in all functions to convert results to Selectors in bulk\"\"\"\n        if not result:\n            return Selectors()\n\n        return self.__elements_convertor(result)\n\n    def __getstate__(self) -> Any:\n        # lxml don't like it :)\n        raise TypeError(\"Can't pickle Selector objects\")\n\n    # The following four properties I made them into functions instead of variables directly\n    # So they don't slow down the process of initializing many instances of the class and gets executed only\n    # when the user needs them for the first time for that specific element and gets cached for next times\n    # Doing that only made the library performance test sky rocked multiple times faster than before\n    # because I was executing them on initialization before :))\n    @property\n    def tag(self) -> str:\n        \"\"\"Get the tag name of the element\"\"\"\n        if self._is_text_node(self._root):\n            return \"#text\"\n        if not self.__tag:\n            self.__tag = str(self._root.tag)\n        return self.__tag or \"\"\n\n    @property\n    def text(self) -> TextHandler:\n        \"\"\"Get text content of the element\"\"\"","sourceCodeStart":234,"sourceCodeEnd":270,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/parser.py#L234-L270","documentation":"Raised by Selector.__getstate__ whenever something attempts to pickle a Selector (pickle.dump, copy.deepcopy, multiprocessing queues, joblib, cloudpickle). The underlying lxml HtmlElement tree cannot be serialized, so the class deliberately fails fast with a TypeError instead of producing a corrupt pickle.","triggerScenarios":"Calling pickle.dumps(selector), copy.deepcopy(selector), passing Selector objects between multiprocessing.Process workers, or using libraries like joblib/pandas that pickle their inputs.","commonSituations":"Scaling a scraper with multiprocessing and sending Selector results through a Pool; caching parsed pages with pickle; deepcopy of a dict that contains Selector objects.","solutions":["Extract plain data (strings, dicts) from the Selector before pickling, e.g. store element.html, element.attrib, or text content instead of the Selector itself.","If multiprocessing is only used for CPU parallelism, switch to threading/asyncio since parsing is C-bound (lxml releases the GIL) and objects never need to cross process boundaries.","As a last resort for process transfer, transfer the raw HTML bytes and re-parse with Selector(content) in the worker."],"exampleFix":"// before\nimport pickle\npickle.dumps(page.css_first('h1'))  # TypeError\n\n// after\nimport pickle\ndata = page.css_first('h1').html  # str is picklable\npickle.dumps(data)","handlingStrategy":"validation","validationCode":"def picklable_extraction(sel):\n    # reduce to primitives before any pickle/deepcopy boundary\n    return {\n        'tag': sel.tag,\n        'text': sel.text,\n        'attrib': dict(sel.attrib),\n        'html': sel.html,\n    }\n\nresults = [picklable_extraction(s) for s in page.css('a')]\npickle.dumps(results)  # safe","typeGuard":"from scrapling.parser import Selector\n\ndef is_selector(obj) -> bool:\n    # true means: do NOT pickle obj directly; extract data first\n    return isinstance(obj, Selector)","tryCatchPattern":"try:\n    pickle.dumps(obj)\nexcept TypeError as e:\n    if \"Can't pickle Selector\" in str(e):\n        raise ValueError('Extract primitive data from Selector before pickling') from e\n    raise","preventionTips":["Never pass Selector objects across multiprocessing boundaries; ship raw HTML or extracted primitives.","Strip Selectors out of any object you intend to cache with pickle/joblib before storing.","Keep a project rule: parsing happens in-process, only str/dict/bytes cross process or cache boundaries."],"tags":["python","pickle","serialization","multiprocessing","lxml"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}