{"record":{"id":"89446111c0f10141","repo":"projectdiscovery/katana","slug":"could-not-get-page-state","errorCode":null,"errorMessage":"could not get page state","messagePattern":"could not get page state","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/engine/headless/crawler/state.go","lineNumber":60,"sourceCode":"\t\t\t\tslog.String(\"current_hash\", currentPageHash),\n\t\t\t\tslog.String(\"origin_hash\", action.OriginID),\n\t\t\t\tslog.Uint64(\"simhash_distance\", uint64(distance)),\n\t\t\t)\n\t\t\t// Treat this page as the origin state to avoid creating a new vertex\n\t\t\treturn originPageState.UniqueID, pageState, nil\n\t\t}\n\t}\n\n\treturn \"\", pageState, fmt.Errorf(\"failed to navigate back to origin page: %s != %s\", currentPageHash, action.OriginID)\n}\n\nfunc getPageHash(page *browser.BrowserPage) (string, *types.PageState, error) {\n\tpageState, err := newPageState(page, nil)\n\tif err == ErrEmptyPage {\n\t\treturn emptyPageHash, nil, nil\n\t}\n\tif err != nil {\n\t\treturn \"\", nil, errors.Wrap(err, \"could not get page state\")\n\t}\n\treturn pageState.UniqueID, pageState, nil\n}\n\nvar ErrEmptyPage = errors.New(\"page is empty\")\n\nfunc newPageState(page *browser.BrowserPage, action *types.Action) (*types.PageState, error) {\n\tpageInfo, err := page.Info()\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"could not get page info\")\n\t}\n\tif pageInfo.URL == \"\" || pageInfo.URL == \"about:blank\" {\n\t\treturn nil, ErrEmptyPage\n\t}\n\n\touterHTML, err := page.HTML()\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"could not get html content\")","sourceCodeStart":42,"sourceCodeEnd":78,"githubUrl":"https://github.com/projectdiscovery/katana/blob/e3e742739c3746f085943ce918fb4e2b8daf6fe6/pkg/engine/headless/crawler/state.go#L42-L78","documentation":"Wrapper from getPageHash when newPageState fails for any reason other than ErrEmptyPage. newPageState gathers page info (page.Info), outer HTML (page.HTML) and the stripped/normalized DOM (domNormalizer.Apply); any of those failing — wrapped as \"could not get page info\", \"could not get html content\", or \"could not get stripped dom\" — surfaces as \"could not get page state\". This breaks both crawlFn state capture and isCorrectNavigation, so the crawler cannot confirm page transitions.","triggerScenarios":"1) page.Info() or page.HTML() CDP calls fail (tab closed, context destroyed mid-navigation). 2) The normalizer fails on the page HTML (see \"failed to apply DOM normalizer\" / \"failed to strip text content\"). 3) Page URL is empty/about:blank is handled separately (ErrEmptyPage) and does NOT produce this error. 4) Renderer crash under load.","commonSituations":"Racing navigation vs. state capture on fast-redirecting sites; normalizer choking on malformed HTML; browser instability during long or highly concurrent crawls.","solutions":["Wait for stable page load before capturing state to avoid context races","Check the wrapped cause (%+v) — fix the underlying Info/HTML/normalizer failure specifically","Retry state capture once; transient CDP errors resolve after load settles","Log and skip the page/transition instead of aborting the entire crawl"],"exampleFix":"// before\nhash, state, err := getPageHash(page)\nif err != nil { return err }\n// after\nhash, state, err := getPageHash(page)\nif err != nil {\n    logger.Debug(\"page state unavailable, skipping transition\", slog.String(\"error\", err.Error()))\n    return nil // or retry once before giving up\n}","handlingStrategy":"retry","validationCode":"// ensure the page is in a capturable state first\ninfo, err := page.Info()\nif err != nil || info.URL == \"\" || info.URL == \"about:blank\" {\n    return // empty or dead page; skip, not retry\n}","typeGuard":"func stateCapturable(p *browser.BrowserPage) bool {\n    info, err := p.Info()\n    return err == nil && info.URL != \"\" && info.URL != \"about:blank\"\n}","tryCatchPattern":"hash, state, err := getPageHash(page)\nif err != nil {\n    if retryable(err) { // CDP context/transport errors\n        time.Sleep(250 * time.Millisecond)\n        hash, state, err = getPageHash(page)\n    }\n    if err != nil { return skipTransition(err) }\n}","preventionTips":["Wait for navigation to settle before hashing state","Distinguish ErrEmptyPage (expected) from real failures","Retry once on transient CDP errors","Check wrapped causes to target the failing step (info/html/normalizer)"],"tags":["headless-browser","page-state","crawl"],"backgroundTag":"cdp-evaluation-failed","analyzedSha":"e3e742739c3746f085943ce918fb4e2b8daf6fe6","analyzedAt":"2026-09-03T14:55:13.248Z","contentChangedAt":"2026-09-03T14:55:13.248Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}