{"record":{"id":"263f6df5e6eab765","repo":"scrapy/scrapy","slug":"no-form-element-found-in-response","errorCode":null,"errorMessage":"No <form> element found in {response}","messagePattern":"No <form> element found in (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scrapy/http/request/form.py","lineNumber":184,"sourceCode":"        (to_bytes(k, enc), to_bytes(v, enc))\n        for k, vs in seq\n        for v in (vs if is_listlike(vs) else [cast(\"str\", vs)])\n    ]\n    return urlencode(values, doseq=True)\n\n\ndef _get_form(\n    response: TextResponse,\n    formname: str | None,\n    formid: str | None,\n    formnumber: int,\n    formxpath: str | None,\n) -> FormElement:\n    \"\"\"Find the wanted form element within the given response.\"\"\"\n    root = response.selector.root\n    forms = root.xpath(\"//form\")\n    if not forms:\n        raise ValueError(f\"No <form> element found in {response}\")\n\n    if formname is not None:\n        f = root.xpath(f'//form[@name=\"{formname}\"]')\n        if f:\n            return cast(\"FormElement\", f[0])\n\n    if formid is not None:\n        f = root.xpath(f'//form[@id=\"{formid}\"]')\n        if f:\n            return cast(\"FormElement\", f[0])\n\n    # Get form element from xpath, if not found, go up\n    if formxpath is not None:\n        nodes = root.xpath(formxpath)\n        if nodes:\n            el = nodes[0]\n            while True:\n                if el.tag == \"form\":","sourceCodeStart":166,"sourceCodeEnd":202,"githubUrl":"https://github.com/scrapy/scrapy/blob/06af687662112027b4482d31e2714a3cf280a91f/scrapy/http/request/form.py#L166-L202","documentation":"_get_form() raises ValueError when the response contains no <form> element at all (root.xpath('//form') is empty). It backs FormRequest.from_response(); before any formname/formid/formxpath/formnumber matching can happen, at least one form must exist, so a form-less page (JS-rendered form, wrong URL, login moved) fails immediately.","triggerScenarios":"FormRequest.from_response(response) on a page whose form is injected by JavaScript (plain Scrapy does not execute JS); response is an error/login-redirect page with no forms; parsing a page whose forms live inside an <iframe> document (not in the main response body); Content-Type or encoding issues leaving the body unparsed as expected.","commonSituations":"Login endpoints changed or now require JS; the target page redirected to a captcha/consent page; site moved form into a frame loaded by a second request; response.follow to the wrong URL due to relative-link mistakes.","solutions":["Inspect the response first: log response.css('form').get() and response.url to confirm what actually came back","If the form is JS-built, use scrapy-playwright or splash to render, or reverse-engineer the underlying POST endpoint and send a plain FormRequest directly","Handle redirects/logins: check response.status and follow to the real form page before calling from_response","Catch ValueError around from_response as a guard so one bad page does not kill the spider"],"exampleFix":"# before\nyield FormRequest.from_response(response, formdata={'user': 'u', 'pass': 'p'})\n# page has no <form> -> ValueError\n\n# after\nif response.css('form'):\n    yield FormRequest.from_response(response, formdata={'user': 'u', 'pass': 'p'})\nelse:\n    self.logger.warning('No form at %s', response.url)","handlingStrategy":"validation","validationCode":"def has_form(response) -> bool:\n    return bool(response.css('form'))\n\nif has_form(response):\n    yield FormRequest.from_response(response, formdata={...})\nelse:\n    self.logger.warning('no form on %s', response.url)","typeGuard":"null","tryCatchPattern":"try:\n    req = FormRequest.from_response(response, formdata=data)\nexcept ValueError as e:\n    self.logger.warning('form extraction failed on %s: %s', response.url, e)\n    return","preventionTips":["Check response.css('form') and response.status before from_response","Beware JS-rendered forms; use scrapy-playwright/splash or the raw POST endpoint","Log the response body/url when login scraping breaks so form presence is diagnosable"],"tags":["forms","from-response","parsing","javascript"],"backgroundTag":null,"analyzedSha":"06af687662112027b4482d31e2714a3cf280a91f","analyzedAt":"2026-08-15T00:21:48.041Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}