projectdiscovery/katana · error

failed to strip text content

Error message

failed to strip text content

What it means

Runtime error from Normalizer.Apply: stripTextContent(firstpass) failed. stripTextContent parses the DOM-normalized HTML with goquery and removes text nodes from content elements (h1..h6, p, span, div, td, th, li, a); errors come from goquery.NewDocumentFromReader (reader issue) or doc.Html() serialization failure. This pass is what makes page hashes stable across text changes.

Source

Thrown at pkg/engine/headless/crawler/normalizer/normalizer.go:52

}

// Apply applies the normalizers to the given content
//
// It normalizes the given content by:
// - Applying the DOM normalizer
// - Applying the text normalizer
// - Denormalizing it
func (n *Normalizer) Apply(text string) (string, error) {
	first := normalizeDocument(text)

	firstpass, err := n.dom.Apply(first)
	if err != nil {
		return "", errors.Wrap(err, "failed to apply DOM normalizer")
	}

	secondpass, err := stripTextContent(firstpass)
	if err != nil {
		return "", errors.Wrap(err, "failed to strip text content")
	}

	thirdpass := n.text.Apply(secondpass)

	fourthpass := normalizeDocument(thirdpass)
	return fourthpass, nil
}

// normalizeDocument normalizes the given document by:
// - Lowercasing it
// - URL decoding it
// - HTML entity decoding it
// - Replacing all whitespace variations with a space
// - Trimming the document whitespaces
func normalizeDocument(text string) string {
	// Lowercase the document
	lowercased := strings.ToLower(text)

View on GitHub (pinned to e3e742739c)

Solutions

1. Update katana to get the latest goquery/parser handling fixes
2. Check the wrapped error to identify whether parse or serialize failed
3. Skip or hash the unstripped DOM as a fallback for pages that fail stripping
4. Bound page size before normalization to avoid pathological parser input

// quick parse probe mirroring stripTextContent
if _, err := goquery.NewDocumentFromReader(strings.NewReader(html)); err != nil {
    return fallback // cannot be stripped
}

func strippable(html string) bool {
    _, err := goquery.NewDocumentFromReader(strings.NewReader(html))
    return err == nil
}

norm, err := normalizer.Apply(html)
if err != nil {
    logger.Debug("strip failed; hashing raw html", slog.String("error", err.Error()))
    norm = html
}

When it happens

Trigger: 1) goquery.NewDocumentFromReader errors parsing the first-pass HTML. 2) doc.Html() fails to serialize the modified document. 3) Extremely large or pathological documents exhausting the parser. 4) Prior passes produced input the parser rejects (nested transformation bug).

Common situations: Crawling huge or deeply nested pages; sites emitting binary garbage misdetected as HTML; katana versions with stripTextContent edge-case bugs.

Related errors

• failed to apply DOM normalizer
• failed to create domnormalizer

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/9f6452dd000982ce. Report an issue: GitHub.