{"record":{"id":"81b33cabbbb93a11","repo":"projectdiscovery/katana","slug":"could-not-get-stripped-dom","errorCode":null,"errorMessage":"could not get stripped dom","messagePattern":"could not get stripped dom","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/engine/headless/crawler/state.go","lineNumber":92,"sourceCode":"\tstrippedDOM, err := getStrippedDOM(outerHTML)\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"could not get stripped dom\")\n\t}\n\tstate.StrippedDOM = strippedDOM\n\n\t// Get sha256 hash of the stripped dom\n\tstate.UniqueID = sha256Hash(strippedDOM)\n\tstate.SimHash = simhash.Fingerprint(strings.NewReader(strippedDOM), 3)\n\n\treturn state, nil\n}\n\nfunc sha256Hash(item string) string {\n\thasher := sha256.New()\n\thasher.Write([]byte(item))\n\thashItem := hex.EncodeToString(hasher.Sum(nil))\n\treturn hashItem\n}\n\nfunc getStrippedDOM(contents string) (string, error) {\n\tnormalized, err := domNormalizer.Apply(contents)\n\tif err != nil {\n\t\treturn \"\", errors.Wrap(err, \"could not normalize dom\")\n\t}\n\treturn normalized, nil\n}","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/projectdiscovery/katana/blob/e3e742739c3746f085943ce918fb4e2b8daf6fe6/pkg/engine/headless/crawler/state.go#L74-L110","documentation":"newPageState fails because getStrippedDOM could not produce a normalized DOM from the page's outer HTML. The error is wrapped from domNormalizer.Apply inside getStrippedDOM and aborts building the PageState, so no UniqueID/SimHash can be computed for deduplication.","triggerScenarios":"Thrown at pkg/engine/headless/crawler/state.go:92 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Inspect the wrapped domNormalizer.Apply error to find the malformed input HTML","Fall back to using the raw outerHTML as the StrippedDOM so crawling can continue","Log the page URL alongside the error to identify offending sites","Add defensive handling in the normalizer for unexpected DOM shapes instead of failing"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"e3e742739c3746f085943ce918fb4e2b8daf6fe6","analyzedAt":"2026-09-03T14:55:13.248Z","contentChangedAt":"2026-09-03T14:55:13.248Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}