D4Vinci/Scrapling · error · ValueError

Selector class needs HTML content, or root arguments to work

Error message

Selector class needs HTML content, or root arguments to work

What it means

ValueError from the Selector (`__init__` in scrapling/parser.py): a Selector must be constructed from HTML content or a prebuilt lxml root element. When both `content` and `root` are None the class has nothing to parse and refuses construction immediately.

Source

Thrown at scrapling/parser.py:119

        It's an old issue with lxml, see `this entry <https://bugs.launchpad.net/lxml/+bug/736708>`

        :param content: HTML content as either string or bytes.
        :param url: It allows storing a URL with the HTML data for retrieving later.
        :param encoding: The encoding type that will be used in HTML parsing, default is `UTF-8`
        :param huge_tree: Enabled by default, should always be enabled when parsing large HTML documents. This controls
             the libxml2 feature that forbids parsing certain large documents to protect from possible memory exhaustion.
        :param root: Used internally to pass etree objects instead of text/body arguments, it takes the highest priority.
            Don't use it unless you know what you are doing!
        :param keep_comments: While parsing the HTML body, drop comments or not. Disabled by default for obvious reasons
        :param keep_cdata: While parsing the HTML body, drop cdata or not. Disabled by default for cleaner HTML.
        :param adaptive: Globally turn off the adaptive feature in all functions, this argument takes higher
            priority over all adaptive related arguments/functions in the class.
        :param storage: The storage class to be passed for adaptive functionalities, see ``Docs`` for more info.
        :param storage_args: A dictionary of ``argument->value`` pairs to be passed for the storage class.
            If empty, default values will be used.
        """
        if root is None and content is None:
            raise ValueError("Selector class needs HTML content, or root arguments to work")

        self.url = url
        self._raw_body: str | bytes = ""
        self.encoding = encoding
        self.__keep_cdata = keep_cdata
        self.__huge_tree_enabled = huge_tree
        self.__keep_comments = keep_comments
        # For selector stuff
        self.__text: Optional[TextHandler] = None
        self.__attributes: Optional[AttributesHandler] = None
        self.__tag: Optional[str] = None
        self._storage: Optional[StorageSystemMixin] = None
        if root is None:
            body: str | bytes
            if isinstance(content, str):
                body = content.strip().replace("\x00", "") or "<html/>"
            elif isinstance(content, bytes):
                body = content.replace(b"\x00", b"")

View on GitHub (pinned to 5d213a2d47)

Solutions

Pass HTML: Selector('<html>...</html>') or Selector(content=response.content or '<html/>').
Guard upstream: if not content: skip/handle the empty case before constructing.
If you already have an lxml element, pass it as root=element.

Example fix

# before
sel = Selector(content=response.content)  # content is None

# after
body = response.content or '<html/>'
if body is None:
    return []
sel = Selector(content=body)

Defensive patterns

Strategy: validation

Validate before calling

body = response.content if response is not None else None
if body is None and root is None:
    raise ValueError('nothing to parse: fetch returned no content')
sel = Selector(content=body, root=root)

Type guard

def has_parse_input(content, root) -> bool:
    return content is not None or root is not None

Try / catch

try:
    sel = Selector(content=body)
except ValueError as e:
    if "needs HTML content" in str(e):
        sel = Selector(content='<html/>')  # explicit empty document
    else:
        raise

Prevention

Check fetch results for None/empty bodies before constructing Selectors.
Default empty pages to '<html/>' deliberately if you must parse on.
Keep parameter names straight: content for HTML, root for lxml elements.

When it happens

Trigger: Selector() with no args; Selector(content=None) when a fetch returned nothing (e.g. empty page or a failed request piped straight in); Selector(root=None, content=None) after a conversion step returned None.

Common situations: Chaining Selector(response.content) where response.content is None on error responses; building Selectors in a loop where some items have no body; refactoring code that used to pass text= (old API) so content is accidentally dropped.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/522576655477d5d7. Report an issue: GitHub.