vxcontrol/pentagi · error · FatalError

internal engine: no readable content found

Error message

internal engine: no readable content found

What it means

Thrown in `internalEngine.analyze` (backend/pkg/tools/searchers/internal.go:172) when zero pages produced usable content but the individual fetches did NOT error — each `FetchMarkdown` succeeded yet returned empty/whitespace-only markdown. Because nothing technically failed, this is a Fatal (not Retryable) error: the engine concludes the discovered pages simply contain no extractable text.

Source

Thrown at backend/pkg/tools/searchers/internal.go:172

			continue
		}
		if len(md) > maxBytes {
			md = md[:maxBytes]
		}
		// The source id (1-based, in fetch order) is what the summarizer cites as
		// [Source N]; usedURLs tracks the matching URL so it can be listed at the end.
		blocks = append(blocks, fmt.Sprintf("<source id=\"%d\" url=\"%s\">\n%s\n</source>", fetched+1, u, md))
		usedURLs = append(usedURLs, u)
		fetched++
	}

	if fetched == 0 {
		// Nothing could be read. If the pages errored (vs. returned empty), surface a
		// retryable error so the orchestrator can fall back to another engine.
		if lastFetch != nil {
			return "", Retryable(fmt.Errorf("internal engine: all pages failed to fetch: %w", lastFetch), 0)
		}
		return "", Fatal(fmt.Errorf("internal engine: no readable content found"))
	}

	prompt := e.buildPrompt(req.Query, blocks)
	answer, serr := e.summarizer(ctx, prompt)
	if serr != nil {
		return "", Retryable(fmt.Errorf("internal engine: summarization failed: %w", serr), 0)
	}

	return appendSources(answer, usedURLs), nil
}

// appendSources appends the list of source URLs the answer was synthesized from, so the
// [Source N] citations in the summary can be traced back to the page they came from. The
// numbering matches the source ids fed to the summarizer (1-based, in fetch order).
func appendSources(answer string, urls []string) string {
	if len(urls) == 0 {
		return answer
	}

View on GitHub (pinned to ea665308ba)

Solutions

1. Confirm the discovered URLs are HTML pages with static text (open them in a browser/view-source)
2. Use a fetcher that executes JavaScript or a rendering proxy for SPA targets
3. Accept the Fatal result and ensure the orchestrator falls back to a non-fetch-based engine (e.g. Perplexity) in web_search.go fallbackStrategy
4. Rephrase the query to surface text-based results pages

// quick sanity check that a discovered URL yields static text
resp, _ := http.Get(u)
ct := resp.Header.Get("Content-Type")
usable := strings.Contains(ct, "text/html") // skip PDFs/images

_, err := engine.Handle(ctx, req)
if err != nil && searchers.IsFatal(err) {
    // do not retry; switch to a different engine immediately
}

When it happens

Trigger: All discovered URLs fetch successfully but return empty markdown — pages that are pure JavaScript-rendered SPAs (no static text), PDFs or images the fetcher cannot convert, or pages that return 200 with an empty body.

Common situations: Fetching modern JS-heavy sites whose content loads client-side; link results pointing to media/PDF files; scraper returning empty bodies for pages requiring cookies/headers.

Related errors

• there are temporarily offline for maintenance. please try ag
• failed to create search log: %w
• failed to get pentester handler: %w
• failed to get searcher handler: %w
• invalid SearchEngineType: %s

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/300405567efdbf1a. Report an issue: GitHub.