vxcontrol/pentagi · error · RetryableError

internal engine: all pages failed to fetch: %w

Error message

internal engine: all pages failed to fetch: %w

What it means

After the internal engine discovers URLs, it fetches each page as markdown (`e.fetcher.FetchMarkdown`). If every page fetch fails (`fetched == 0`) and at least one fetch returned an actual error (`lastFetch != nil`), it returns this Retryable error wrapping the last fetch failure (backend/pkg/tools/searchers/internal.go:170). Retryable tells the orchestrator the whole internal engine may be retried or another engine tried — the failure is transient, not a query problem.

Source

Thrown at backend/pkg/tools/searchers/internal.go:170

		md = strings.TrimSpace(md)
		if md == "" {
			continue
		}
		if len(md) > maxBytes {
			md = md[:maxBytes]
		}
		// The source id (1-based, in fetch order) is what the summarizer cites as
		// [Source N]; usedURLs tracks the matching URL so it can be listed at the end.
		blocks = append(blocks, fmt.Sprintf("<source id=\"%d\" url=\"%s\">\n%s\n</source>", fetched+1, u, md))
		usedURLs = append(usedURLs, u)
		fetched++
	}

	if fetched == 0 {
		// Nothing could be read. If the pages errored (vs. returned empty), surface a
		// retryable error so the orchestrator can fall back to another engine.
		if lastFetch != nil {
			return "", Retryable(fmt.Errorf("internal engine: all pages failed to fetch: %w", lastFetch), 0)
		}
		return "", Fatal(fmt.Errorf("internal engine: no readable content found"))
	}

	prompt := e.buildPrompt(req.Query, blocks)
	answer, serr := e.summarizer(ctx, prompt)
	if serr != nil {
		return "", Retryable(fmt.Errorf("internal engine: summarization failed: %w", serr), 0)
	}

	return appendSources(answer, usedURLs), nil
}

// appendSources appends the list of source URLs the answer was synthesized from, so the
// [Source N] citations in the summary can be traced back to the page they came from. The
// numbering matches the source ids fed to the summarizer (1-based, in fetch order).
func appendSources(answer string, urls []string) string {
	if len(urls) == 0 {

View on GitHub (pinned to ea665308ba)

Solutions

1. Verify outbound network/DNS/proxy from the container running PentAGI (curl one of the discovered URLs directly)
2. Check the wrapped `lastFetch` error in the message — it names the concrete fetch problem (timeout, 403, TLS) and fix that cause
3. Raise fetch timeout or configure the fetcher's proxy/user-agent settings so bot-protected sites can be retrieved
4. Let the orchestrator fall back to another search engine (this error is Retryable by design); ensure a fallback engine is configured in web_search.go

Example fix

// before
md, ferr := e.fetcher.FetchMarkdown(ctx, u) // all fail: no proxy configured
// after: set proxy env before starting
export HTTP_PROXY=http://proxy:3128
export HTTPS_PROXY=http://proxy:3128

// verify egress before enabling internal engine
cmd := exec.CommandContext(ctx, "curl", "-fsS", "-o", "/dev/null", "https://example.com")
if err := cmd.Run(); err != nil { /* no outbound network */ }

result, err := engine.Handle(ctx, req)
if err != nil {
    if searchers.IsRetryable(err) {
        // safe to retry after backoff, or fall back to another engine
    }
}

When it happens

Trigger: `analyze` loops over discovered URLs; every `FetchMarkdown(ctx, u)` call errors (network failure, DNS failure, target blocks the fetcher, TLS error, timeouts) so `fetched` stays 0 while `lastFetch` holds the final error.

Common situations: Container has no outbound internet access or broken DNS; all candidate pages sit behind bot protection (Cloudflare 403); the corporate proxy is down; target sites time out under load.

Related errors

• there are temporarily offline for maintenance. please try ag
• failed to create search log: %w
• failed to generate subtasks for task %d: %w
• knowledge: embed query: %w
• knowledge: compute embedding: %w

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/a90c2601253d101e. Report an issue: GitHub.