vxcontrol/pentagi · error · FatalError

internal engine: link discovery returned no usable URLs

Error message

internal engine: link discovery returned no usable URLs

What it means

The internal search engine of PentAGI works in two stages: it first asks a configured link-discovery engine (e.g. DuckDuckGo) for candidate URLs, then fetches and summarizes the pages. This error is thrown in `internalEngine.analyze` (backend/pkg/tools/searchers/internal.go:119) when `discoverURLs` succeeded but the URL-extraction regex found zero usable URLs in the engine's markdown output. It is wrapped as a Fatal typed error, meaning the orchestrator should fall back to another search engine rather than retry the same path.

Source

Thrown at backend/pkg/tools/searchers/internal.go:119

				"engine": "internal",
				"query":  req.Query,
				"error":  err.Error(),
			}),
		)
		obs.LogErrorOrCancel(logger, err, "internal analytics engine failed")
		return "", err
	}

	return result, nil
}

func (e *internalEngine) analyze(ctx context.Context, req Request) (string, error) {
	urls, out, err := e.discoverURLs(ctx, req)
	if err != nil {
		return "", err
	}
	if len(urls) == 0 {
		return "", Fatal(fmt.Errorf("internal engine: link discovery returned no usable URLs"))
	}

	maxSites := e.cfg.WebSearchInternalMaxSites
	if maxSites <= 0 {
		maxSites = internalDefaultMaxSites
	}
	maxBytes := e.cfg.WebSearchInternalMaxSiteBytes
	if maxBytes <= 0 {
		maxBytes = internalDefaultMaxSiteBytes
	}

	var (
		blocks    []string
		usedURLs  []string
		fetched   int
		lastFetch error
	)
	out = strings.TrimSpace(out)

View on GitHub (pinned to ea665308ba)

Solutions

1. Re-run the query with different, more common keywords to get real results from the link-discovery stage
2. Check the upstream link engine's health/availability (rate limits, blocked IP, expired API key) and fix that first; the internal engine only reflects what discovery returns
3. Configure an additional link-discovery engine in the fallbackStrategy chain in web_search.go so discovery has a second source
4. If the upstream format changed, update `urlPattern` in internal.go to match the new output format

Example fix

// before: single discovery engine, empty result is fatal
links := []searchers.Searcher{duckduckgo}

// after: add a fallback discovery engine
links := []searchers.Searcher{duckduckgo, brave} // configured in buildSearchEngines/fallbackStrategy

// pre-check discovery output before treating internal engine as authoritative
urls := urlPattern.FindAllString(out, -1)
if len(dedupeURLs(urls)) == 0 {
    // route to another engine instead
}

// typed-error check (Go)
result, err := engine.Handle(ctx, req)
if err != nil {
    var f *searchers.FatalError
    if errors.As(err, &f) {
        // skip retries; try next engine in fallbackStrategy
    }
}

When it happens

Trigger: `Handle(ctx, req)` on the internal engine is called, the delegated link engine returns a result string, but `urlPattern.FindAllString(out, -1)` matches no URLs — e.g. the upstream engine returned an empty result set for a very obscure query, returned only prose with no links, or the output format changed so the regex no longer matches.

Common situations: Querying extremely niche terms that yield zero web results; a link-discovery engine (DuckDuckGo etc.) silently degrading and returning an HTML block page or rate-limit notice instead of results; upgrading the upstream engine so its output layout no longer contains bare URLs the `urlPattern` regex can see.

Related errors

• there are temporarily offline for maintenance. please try ag
• failed to create search log: %w
• no subtasks found for task %d: %w
• knowledge: embedder returned no vectors for query
• knowledge: embedder returned no vectors

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/7f2b35820ea8f683. Report an issue: GitHub.