What causes "failed to apply DOM normalizer" in projectdiscovery/katana?

Inspect the wrapped cause to see whether the DOM parser rejects the document

failed to apply DOM normalizer

Error message

failed to apply DOM normalizer

What it means

Runtime error from Normalizer.Apply: the DOM normalization pass (n.dom.Apply on a normalized copy of the document) returned an error, meaning the DOM-based page normalization could not process the HTML. Apply is used to compute stripped DOM / page hashes during crawling, so this error fails page-state creation.

Source

Thrown at pkg/engine/headless/crawler/normalizer/normalizer.go:47

	domNormalizer := NewDOMNormalizer()
	return &Normalizer{
		dom:  domNormalizer,
		text: textNormalizer,
	}, nil
}

// Apply applies the normalizers to the given content
//
// It normalizes the given content by:
// - Applying the DOM normalizer
// - Applying the text normalizer
// - Denormalizing it
func (n *Normalizer) Apply(text string) (string, error) {
	first := normalizeDocument(text)

	firstpass, err := n.dom.Apply(first)
	if err != nil {
		return "", errors.Wrap(err, "failed to apply DOM normalizer")
	}

	secondpass, err := stripTextContent(firstpass)
	if err != nil {
		return "", errors.Wrap(err, "failed to strip text content")
	}

	thirdpass := n.text.Apply(secondpass)

	fourthpass := normalizeDocument(thirdpass)
	return fourthpass, nil
}

// normalizeDocument normalizes the given document by:
// - Lowercasing it
// - URL decoding it
// - HTML entity decoding it
// - Replacing all whitespace variations with a space

View on GitHub (pinned to e3e742739c)

Solutions

Inspect the wrapped cause to see whether the DOM parser rejects the document
Sanitize/repair HTML before normalization (e.g. run a lenient parser pass) if feeding custom content
Update katana — DOMNormalizer fixes land regularly for edge-case HTML
Log the offending page and skip it rather than failing the crawl

Example fix

// before
stripped, err := normalizer.Apply(html)
if err != nil { return err }
// after
stripped, err := normalizer.Apply(html)
if err != nil {
    logger.Debug("normalization failed, using raw html", slog.String("error", err.Error()))
    stripped = html // fallback keeps the crawl going
}

Defensive patterns

Strategy: try-catch

Validate before calling

// guard against obviously unparseable content before normalizing
if strings.TrimSpace(html) == "" { return fallback }

Type guard

func normalizable(html string) bool { return strings.TrimSpace(html) != "" }

Try / catch

stripped, err := n.Apply(doc)
if err != nil {
    logger.Debug("dom normalization failed, using raw dom", slog.String("error", err.Error()))
    stripped = doc // hash on raw content instead of failing
}

Prevention

Sanitize malformed HTML before normalization
Update katana for DOMNormalizer edge-case fixes
Bound document size before parsing
Log and skip pathological pages rather than aborting

When it happens

Trigger: 1) n.dom.Apply fails parsing or transforming the (already whitespace/entity-normalized) HTML string — e.g. malformed or pathological input the DOM normalizer cannot handle. 2) The normalizer instance was mis-constructed upstream (nil/invalid components) so Apply errors. 3) Extremely large documents causing the underlying parse to fail.

Common situations: Crawling pages with heavily malformed or non-standard HTML; documents with broken encodings that survive the first normalizeDocument pass; custom DOMNormalizer implementations erroring on unusual nodes.

Related errors

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/59158fe53b4560cb. Report an issue: GitHub.