{"record":{"id":"a90c2601253d101e","repo":"vxcontrol/pentagi","slug":"internal-engine-all-pages-failed-to-fetch-w","errorCode":null,"errorMessage":"internal engine: all pages failed to fetch: %w","messagePattern":"internal engine: all pages failed to fetch: %w","errorType":"exception","errorClass":"RetryableError","httpStatus":null,"severity":"error","filePath":"backend/pkg/tools/searchers/internal.go","lineNumber":170,"sourceCode":"\t\tmd = strings.TrimSpace(md)\n\t\tif md == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tif len(md) > maxBytes {\n\t\t\tmd = md[:maxBytes]\n\t\t}\n\t\t// The source id (1-based, in fetch order) is what the summarizer cites as\n\t\t// [Source N]; usedURLs tracks the matching URL so it can be listed at the end.\n\t\tblocks = append(blocks, fmt.Sprintf(\"<source id=\\\"%d\\\" url=\\\"%s\\\">\\n%s\\n</source>\", fetched+1, u, md))\n\t\tusedURLs = append(usedURLs, u)\n\t\tfetched++\n\t}\n\n\tif fetched == 0 {\n\t\t// Nothing could be read. If the pages errored (vs. returned empty), surface a\n\t\t// retryable error so the orchestrator can fall back to another engine.\n\t\tif lastFetch != nil {\n\t\t\treturn \"\", Retryable(fmt.Errorf(\"internal engine: all pages failed to fetch: %w\", lastFetch), 0)\n\t\t}\n\t\treturn \"\", Fatal(fmt.Errorf(\"internal engine: no readable content found\"))\n\t}\n\n\tprompt := e.buildPrompt(req.Query, blocks)\n\tanswer, serr := e.summarizer(ctx, prompt)\n\tif serr != nil {\n\t\treturn \"\", Retryable(fmt.Errorf(\"internal engine: summarization failed: %w\", serr), 0)\n\t}\n\n\treturn appendSources(answer, usedURLs), nil\n}\n\n// appendSources appends the list of source URLs the answer was synthesized from, so the\n// [Source N] citations in the summary can be traced back to the page they came from. The\n// numbering matches the source ids fed to the summarizer (1-based, in fetch order).\nfunc appendSources(answer string, urls []string) string {\n\tif len(urls) == 0 {","sourceCodeStart":152,"sourceCodeEnd":188,"githubUrl":"https://github.com/vxcontrol/pentagi/blob/ea665308baaff015b226f308438a68d929d0f29b/backend/pkg/tools/searchers/internal.go#L152-L188","documentation":"After the internal engine discovers URLs, it fetches each page as markdown (`e.fetcher.FetchMarkdown`). If every page fetch fails (`fetched == 0`) and at least one fetch returned an actual error (`lastFetch != nil`), it returns this Retryable error wrapping the last fetch failure (backend/pkg/tools/searchers/internal.go:170). Retryable tells the orchestrator the whole internal engine may be retried or another engine tried — the failure is transient, not a query problem.","triggerScenarios":"`analyze` loops over discovered URLs; every `FetchMarkdown(ctx, u)` call errors (network failure, DNS failure, target blocks the fetcher, TLS error, timeouts) so `fetched` stays 0 while `lastFetch` holds the final error.","commonSituations":"Container has no outbound internet access or broken DNS; all candidate pages sit behind bot protection (Cloudflare 403); the corporate proxy is down; target sites time out under load.","solutions":["Verify outbound network/DNS/proxy from the container running PentAGI (curl one of the discovered URLs directly)","Check the wrapped `lastFetch` error in the message — it names the concrete fetch problem (timeout, 403, TLS) and fix that cause","Raise fetch timeout or configure the fetcher's proxy/user-agent settings so bot-protected sites can be retrieved","Let the orchestrator fall back to another search engine (this error is Retryable by design); ensure a fallback engine is configured in web_search.go"],"exampleFix":"// before\nmd, ferr := e.fetcher.FetchMarkdown(ctx, u) // all fail: no proxy configured\n// after: set proxy env before starting\nexport HTTP_PROXY=http://proxy:3128\nexport HTTPS_PROXY=http://proxy:3128","handlingStrategy":"retry","validationCode":"// verify egress before enabling internal engine\ncmd := exec.CommandContext(ctx, \"curl\", \"-fsS\", \"-o\", \"/dev/null\", \"https://example.com\")\nif err := cmd.Run(); err != nil { /* no outbound network */ }","typeGuard":null,"tryCatchPattern":"result, err := engine.Handle(ctx, req)\nif err != nil {\n    if searchers.IsRetryable(err) {\n        // safe to retry after backoff, or fall back to another engine\n    }\n}","preventionTips":["Ensure containers have working DNS and egress (or a valid proxy) before enabling fetch-based engines","Check the wrapped lastFetch cause — fix the concrete fetch error (403 bot-block, TLS, timeout)","Add a rendering-capable fetcher or user-agent tuning if targets use bot protection","Always configure a non-fetch fallback engine (e.g. Perplexity) in web_search.go"],"tags":["network","fetch","search"],"backgroundTag":"all-requests-failed","analyzedSha":"ea665308baaff015b226f308438a68d929d0f29b","analyzedAt":"2026-09-01T14:16:31.421Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}