projectdiscovery/katana · error

could not get stripped dom

Error message

could not get stripped dom

What it means

newPageState fails because getStrippedDOM could not produce a normalized DOM from the page's outer HTML. The error is wrapped from domNormalizer.Apply inside getStrippedDOM and aborts building the PageState, so no UniqueID/SimHash can be computed for deduplication.

Source

Thrown at pkg/engine/headless/crawler/state.go:92

	strippedDOM, err := getStrippedDOM(outerHTML)
	if err != nil {
		return nil, errors.Wrap(err, "could not get stripped dom")
	}
	state.StrippedDOM = strippedDOM

	// Get sha256 hash of the stripped dom
	state.UniqueID = sha256Hash(strippedDOM)
	state.SimHash = simhash.Fingerprint(strings.NewReader(strippedDOM), 3)

	return state, nil
}

func sha256Hash(item string) string {
	hasher := sha256.New()
	hasher.Write([]byte(item))
	hashItem := hex.EncodeToString(hasher.Sum(nil))
	return hashItem
}

func getStrippedDOM(contents string) (string, error) {
	normalized, err := domNormalizer.Apply(contents)
	if err != nil {
		return "", errors.Wrap(err, "could not normalize dom")
	}
	return normalized, nil
}

View on GitHub (pinned to e3e742739c)

Solutions

1. Inspect the wrapped domNormalizer.Apply error to find the malformed input HTML
2. Fall back to using the raw outerHTML as the StrippedDOM so crawling can continue
3. Log the page URL alongside the error to identify offending sites
4. Add defensive handling in the normalizer for unexpected DOM shapes instead of failing

When it happens

Trigger: Thrown at pkg/engine/headless/crawler/state.go:92 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/81b33cabbbb93a11. Report an issue: GitHub.