{"record":{"id":"59158fe53b4560cb","repo":"projectdiscovery/katana","slug":"failed-to-apply-dom-normalizer","errorCode":null,"errorMessage":"failed to apply DOM normalizer","messagePattern":"failed to apply DOM normalizer","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/engine/headless/crawler/normalizer/normalizer.go","lineNumber":47,"sourceCode":"\tdomNormalizer := NewDOMNormalizer()\n\treturn &Normalizer{\n\t\tdom:  domNormalizer,\n\t\ttext: textNormalizer,\n\t}, nil\n}\n\n// Apply applies the normalizers to the given content\n//\n// It normalizes the given content by:\n// - Applying the DOM normalizer\n// - Applying the text normalizer\n// - Denormalizing it\nfunc (n *Normalizer) Apply(text string) (string, error) {\n\tfirst := normalizeDocument(text)\n\n\tfirstpass, err := n.dom.Apply(first)\n\tif err != nil {\n\t\treturn \"\", errors.Wrap(err, \"failed to apply DOM normalizer\")\n\t}\n\n\tsecondpass, err := stripTextContent(firstpass)\n\tif err != nil {\n\t\treturn \"\", errors.Wrap(err, \"failed to strip text content\")\n\t}\n\n\tthirdpass := n.text.Apply(secondpass)\n\n\tfourthpass := normalizeDocument(thirdpass)\n\treturn fourthpass, nil\n}\n\n// normalizeDocument normalizes the given document by:\n// - Lowercasing it\n// - URL decoding it\n// - HTML entity decoding it\n// - Replacing all whitespace variations with a space","sourceCodeStart":29,"sourceCodeEnd":65,"githubUrl":"https://github.com/projectdiscovery/katana/blob/e3e742739c3746f085943ce918fb4e2b8daf6fe6/pkg/engine/headless/crawler/normalizer/normalizer.go#L29-L65","documentation":"Runtime error from Normalizer.Apply: the DOM normalization pass (n.dom.Apply on a normalized copy of the document) returned an error, meaning the DOM-based page normalization could not process the HTML. Apply is used to compute stripped DOM / page hashes during crawling, so this error fails page-state creation.","triggerScenarios":"1) n.dom.Apply fails parsing or transforming the (already whitespace/entity-normalized) HTML string — e.g. malformed or pathological input the DOM normalizer cannot handle. 2) The normalizer instance was mis-constructed upstream (nil/invalid components) so Apply errors. 3) Extremely large documents causing the underlying parse to fail.","commonSituations":"Crawling pages with heavily malformed or non-standard HTML; documents with broken encodings that survive the first normalizeDocument pass; custom DOMNormalizer implementations erroring on unusual nodes.","solutions":["Inspect the wrapped cause to see whether the DOM parser rejects the document","Sanitize/repair HTML before normalization (e.g. run a lenient parser pass) if feeding custom content","Update katana — DOMNormalizer fixes land regularly for edge-case HTML","Log the offending page and skip it rather than failing the crawl"],"exampleFix":"// before\nstripped, err := normalizer.Apply(html)\nif err != nil { return err }\n// after\nstripped, err := normalizer.Apply(html)\nif err != nil {\n    logger.Debug(\"normalization failed, using raw html\", slog.String(\"error\", err.Error()))\n    stripped = html // fallback keeps the crawl going\n}","handlingStrategy":"try-catch","validationCode":"// guard against obviously unparseable content before normalizing\nif strings.TrimSpace(html) == \"\" { return fallback }","typeGuard":"func normalizable(html string) bool { return strings.TrimSpace(html) != \"\" }","tryCatchPattern":"stripped, err := n.Apply(doc)\nif err != nil {\n    logger.Debug(\"dom normalization failed, using raw dom\", slog.String(\"error\", err.Error()))\n    stripped = doc // hash on raw content instead of failing\n}","preventionTips":["Sanitize malformed HTML before normalization","Update katana for DOMNormalizer edge-case fixes","Bound document size before parsing","Log and skip pathological pages rather than aborting"],"tags":["normalizer","html-parsing","crawl"],"backgroundTag":"html-normalization-failed","analyzedSha":"e3e742739c3746f085943ce918fb4e2b8daf6fe6","analyzedAt":"2026-09-03T14:55:13.248Z","contentChangedAt":"2026-09-03T14:55:13.248Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}