projectdiscovery/katana · error

could not get page state

Error message

could not get page state

What it means

Wrapper from getPageHash when newPageState fails for any reason other than ErrEmptyPage. newPageState gathers page info (page.Info), outer HTML (page.HTML) and the stripped/normalized DOM (domNormalizer.Apply); any of those failing — wrapped as "could not get page info", "could not get html content", or "could not get stripped dom" — surfaces as "could not get page state". This breaks both crawlFn state capture and isCorrectNavigation, so the crawler cannot confirm page transitions.

Source

Thrown at pkg/engine/headless/crawler/state.go:60

				slog.String("current_hash", currentPageHash),
				slog.String("origin_hash", action.OriginID),
				slog.Uint64("simhash_distance", uint64(distance)),
			)
			// Treat this page as the origin state to avoid creating a new vertex
			return originPageState.UniqueID, pageState, nil
		}
	}

	return "", pageState, fmt.Errorf("failed to navigate back to origin page: %s != %s", currentPageHash, action.OriginID)
}

func getPageHash(page *browser.BrowserPage) (string, *types.PageState, error) {
	pageState, err := newPageState(page, nil)
	if err == ErrEmptyPage {
		return emptyPageHash, nil, nil
	}
	if err != nil {
		return "", nil, errors.Wrap(err, "could not get page state")
	}
	return pageState.UniqueID, pageState, nil
}

var ErrEmptyPage = errors.New("page is empty")

func newPageState(page *browser.BrowserPage, action *types.Action) (*types.PageState, error) {
	pageInfo, err := page.Info()
	if err != nil {
		return nil, errors.Wrap(err, "could not get page info")
	}
	if pageInfo.URL == "" || pageInfo.URL == "about:blank" {
		return nil, ErrEmptyPage
	}

	outerHTML, err := page.HTML()
	if err != nil {
		return nil, errors.Wrap(err, "could not get html content")

View on GitHub (pinned to e3e742739c)

Solutions

  1. Wait for stable page load before capturing state to avoid context races
  2. Check the wrapped cause (%+v) — fix the underlying Info/HTML/normalizer failure specifically
  3. Retry state capture once; transient CDP errors resolve after load settles
  4. Log and skip the page/transition instead of aborting the entire crawl

Example fix

// before
hash, state, err := getPageHash(page)
if err != nil { return err }
// after
hash, state, err := getPageHash(page)
if err != nil {
    logger.Debug("page state unavailable, skipping transition", slog.String("error", err.Error()))
    return nil // or retry once before giving up
}
Defensive patterns

Strategy: retry

Validate before calling

// ensure the page is in a capturable state first
info, err := page.Info()
if err != nil || info.URL == "" || info.URL == "about:blank" {
    return // empty or dead page; skip, not retry
}

Type guard

func stateCapturable(p *browser.BrowserPage) bool {
    info, err := p.Info()
    return err == nil && info.URL != "" && info.URL != "about:blank"
}

Try / catch

hash, state, err := getPageHash(page)
if err != nil {
    if retryable(err) { // CDP context/transport errors
        time.Sleep(250 * time.Millisecond)
        hash, state, err = getPageHash(page)
    }
    if err != nil { return skipTransition(err) }
}

Prevention

When it happens

Trigger: 1) page.Info() or page.HTML() CDP calls fail (tab closed, context destroyed mid-navigation). 2) The normalizer fails on the page HTML (see "failed to apply DOM normalizer" / "failed to strip text content"). 3) Page URL is empty/about:blank is handled separately (ErrEmptyPage) and does NOT produce this error. 4) Renderer crash under load.

Common situations: Racing navigation vs. state capture on fast-redirecting sites; normalizer choking on malformed HTML; browser instability during long or highly concurrent crawls.

Related errors

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/89446111c0f10141. Report an issue: GitHub.