{"record":{"id":"7f2b35820ea8f683","repo":"vxcontrol/pentagi","slug":"internal-engine-link-discovery-returned-no-usable","errorCode":null,"errorMessage":"internal engine: link discovery returned no usable URLs","messagePattern":"internal engine: link discovery returned no usable URLs","errorType":"exception","errorClass":"FatalError","httpStatus":null,"severity":"error","filePath":"backend/pkg/tools/searchers/internal.go","lineNumber":119,"sourceCode":"\t\t\t\t\"engine\": \"internal\",\n\t\t\t\t\"query\":  req.Query,\n\t\t\t\t\"error\":  err.Error(),\n\t\t\t}),\n\t\t)\n\t\tobs.LogErrorOrCancel(logger, err, \"internal analytics engine failed\")\n\t\treturn \"\", err\n\t}\n\n\treturn result, nil\n}\n\nfunc (e *internalEngine) analyze(ctx context.Context, req Request) (string, error) {\n\turls, out, err := e.discoverURLs(ctx, req)\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\tif len(urls) == 0 {\n\t\treturn \"\", Fatal(fmt.Errorf(\"internal engine: link discovery returned no usable URLs\"))\n\t}\n\n\tmaxSites := e.cfg.WebSearchInternalMaxSites\n\tif maxSites <= 0 {\n\t\tmaxSites = internalDefaultMaxSites\n\t}\n\tmaxBytes := e.cfg.WebSearchInternalMaxSiteBytes\n\tif maxBytes <= 0 {\n\t\tmaxBytes = internalDefaultMaxSiteBytes\n\t}\n\n\tvar (\n\t\tblocks    []string\n\t\tusedURLs  []string\n\t\tfetched   int\n\t\tlastFetch error\n\t)\n\tout = strings.TrimSpace(out)","sourceCodeStart":101,"sourceCodeEnd":137,"githubUrl":"https://github.com/vxcontrol/pentagi/blob/ea665308baaff015b226f308438a68d929d0f29b/backend/pkg/tools/searchers/internal.go#L101-L137","documentation":"The internal search engine of PentAGI works in two stages: it first asks a configured link-discovery engine (e.g. DuckDuckGo) for candidate URLs, then fetches and summarizes the pages. This error is thrown in `internalEngine.analyze` (backend/pkg/tools/searchers/internal.go:119) when `discoverURLs` succeeded but the URL-extraction regex found zero usable URLs in the engine's markdown output. It is wrapped as a Fatal typed error, meaning the orchestrator should fall back to another search engine rather than retry the same path.","triggerScenarios":"`Handle(ctx, req)` on the internal engine is called, the delegated link engine returns a result string, but `urlPattern.FindAllString(out, -1)` matches no URLs — e.g. the upstream engine returned an empty result set for a very obscure query, returned only prose with no links, or the output format changed so the regex no longer matches.","commonSituations":"Querying extremely niche terms that yield zero web results; a link-discovery engine (DuckDuckGo etc.) silently degrading and returning an HTML block page or rate-limit notice instead of results; upgrading the upstream engine so its output layout no longer contains bare URLs the `urlPattern` regex can see.","solutions":["Re-run the query with different, more common keywords to get real results from the link-discovery stage","Check the upstream link engine's health/availability (rate limits, blocked IP, expired API key) and fix that first; the internal engine only reflects what discovery returns","Configure an additional link-discovery engine in the fallbackStrategy chain in web_search.go so discovery has a second source","If the upstream format changed, update `urlPattern` in internal.go to match the new output format"],"exampleFix":"// before: single discovery engine, empty result is fatal\nlinks := []searchers.Searcher{duckduckgo}\n\n// after: add a fallback discovery engine\nlinks := []searchers.Searcher{duckduckgo, brave} // configured in buildSearchEngines/fallbackStrategy","handlingStrategy":"fallback","validationCode":"// pre-check discovery output before treating internal engine as authoritative\nurls := urlPattern.FindAllString(out, -1)\nif len(dedupeURLs(urls)) == 0 {\n    // route to another engine instead\n}","typeGuard":null,"tryCatchPattern":"// typed-error check (Go)\nresult, err := engine.Handle(ctx, req)\nif err != nil {\n    var f *searchers.FatalError\n    if errors.As(err, &f) {\n        // skip retries; try next engine in fallbackStrategy\n    }\n}","preventionTips":["Configure at least two link-discovery engines in the fallbackStrategy chain","Monitor upstream engines for empty/blocked responses with alerting on search-log engine stats","Keep urlPattern in sync with upstream engine output formats when upgrading"],"tags":["search","url-discovery","empty-result"],"backgroundTag":"search-returned-no-results","analyzedSha":"ea665308baaff015b226f308438a68d929d0f29b","analyzedAt":"2026-09-01T14:16:31.421Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}