{"record":{"id":"9f6452dd000982ce","repo":"projectdiscovery/katana","slug":"failed-to-strip-text-content","errorCode":null,"errorMessage":"failed to strip text content","messagePattern":"failed to strip text content","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/engine/headless/crawler/normalizer/normalizer.go","lineNumber":52,"sourceCode":"}\n\n// Apply applies the normalizers to the given content\n//\n// It normalizes the given content by:\n// - Applying the DOM normalizer\n// - Applying the text normalizer\n// - Denormalizing it\nfunc (n *Normalizer) Apply(text string) (string, error) {\n\tfirst := normalizeDocument(text)\n\n\tfirstpass, err := n.dom.Apply(first)\n\tif err != nil {\n\t\treturn \"\", errors.Wrap(err, \"failed to apply DOM normalizer\")\n\t}\n\n\tsecondpass, err := stripTextContent(firstpass)\n\tif err != nil {\n\t\treturn \"\", errors.Wrap(err, \"failed to strip text content\")\n\t}\n\n\tthirdpass := n.text.Apply(secondpass)\n\n\tfourthpass := normalizeDocument(thirdpass)\n\treturn fourthpass, nil\n}\n\n// normalizeDocument normalizes the given document by:\n// - Lowercasing it\n// - URL decoding it\n// - HTML entity decoding it\n// - Replacing all whitespace variations with a space\n// - Trimming the document whitespaces\nfunc normalizeDocument(text string) string {\n\t// Lowercase the document\n\tlowercased := strings.ToLower(text)\n","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/projectdiscovery/katana/blob/e3e742739c3746f085943ce918fb4e2b8daf6fe6/pkg/engine/headless/crawler/normalizer/normalizer.go#L34-L70","documentation":"Runtime error from Normalizer.Apply: stripTextContent(firstpass) failed. stripTextContent parses the DOM-normalized HTML with goquery and removes text nodes from content elements (h1..h6, p, span, div, td, th, li, a); errors come from goquery.NewDocumentFromReader (reader issue) or doc.Html() serialization failure. This pass is what makes page hashes stable across text changes.","triggerScenarios":"1) goquery.NewDocumentFromReader errors parsing the first-pass HTML. 2) doc.Html() fails to serialize the modified document. 3) Extremely large or pathological documents exhausting the parser. 4) Prior passes produced input the parser rejects (nested transformation bug).","commonSituations":"Crawling huge or deeply nested pages; sites emitting binary garbage misdetected as HTML; katana versions with stripTextContent edge-case bugs.","solutions":["Update katana to get the latest goquery/parser handling fixes","Check the wrapped error to identify whether parse or serialize failed","Skip or hash the unstripped DOM as a fallback for pages that fail stripping","Bound page size before normalization to avoid pathological parser input"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// quick parse probe mirroring stripTextContent\nif _, err := goquery.NewDocumentFromReader(strings.NewReader(html)); err != nil {\n    return fallback // cannot be stripped\n}","typeGuard":"func strippable(html string) bool {\n    _, err := goquery.NewDocumentFromReader(strings.NewReader(html))\n    return err == nil\n}","tryCatchPattern":"norm, err := normalizer.Apply(html)\nif err != nil {\n    logger.Debug(\"strip failed; hashing raw html\", slog.String(\"error\", err.Error()))\n    norm = html\n}","preventionTips":["Keep goquery/katana versions current","Limit huge pages before normalization","Fallback to unstripped hashing when stripping fails","Watch for documents that fail parse vs serialize and file issues accordingly"],"tags":["normalizer","goquery","html-parsing"],"backgroundTag":"html-normalization-failed","analyzedSha":"e3e742739c3746f085943ce918fb4e2b8daf6fe6","analyzedAt":"2026-09-03T14:55:13.248Z","contentChangedAt":"2026-09-03T14:55:13.248Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}