Tencent/WeKnora · error

duckduckgo HTML returned status %d

Error message

duckduckgo HTML returned status %d

What it means

DuckDuckGo's HTML endpoint returned a status other than 200 or 202, so searchHTML rejects it with this formatted status code error. It means the scrape request reached DuckDuckGo but was not served the normal results page.

Source

Thrown at internal/infrastructure/web_search/duckduckgo.go:100

	req.Header.Set(
		"User-Agent",
		"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
	)

	curlCommand := fmt.Sprintf(
		"curl -X GET '%s' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'",
		req.URL.String(),
	)
	logger.Infof(ctx, "Curl of request: %s", secutils.SanitizeForLog(curlCommand))

	resp, err := p.client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("failed to perform request: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted {
		return nil, fmt.Errorf("duckduckgo HTML returned status %d", resp.StatusCode)
	}

	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("failed to parse HTML: %w", err)
	}

	results := make([]*types.WebSearchResult, 0, maxResults)
	doc.Find(".web-result").Each(func(i int, s *goquery.Selection) {
		if len(results) >= maxResults {
			return
		}
		titleNode := s.Find(".result__a")
		title := strings.TrimSpace(titleNode.Text())
		var link string
		if href, exists := titleNode.Attr("href"); exists {
			link = cleanDDGURL(href)
		}

View on GitHub (pinned to 988cbb0330)

Solutions

1. Check the status code in the message: 202/403 usually means bot detection — slow down, rotate IPs/proxies, keep headers consistent.
2. 429 -> back off significantly and retry later.
3. 5xx -> retry with backoff; may be transient DuckDuckGo outage.
4. Rely on the Instant Answer API fallback or switch providers for programmatic access.

func isDDGStatusErr(err error) (int, bool) {
    if err == nil { return 0, false }
    var status int
    if _, scanErr := fmt.Sscanf(err.Error(), "duckduckgo HTML returned status %d", &status); scanErr == nil {
        return status, true
    }
    return 0, false
}

results, err := ddg.Search(ctx, query, 10, false)
if err != nil {
    if strings.Contains(err.Error(), "duckduckgo HTML returned status") {
        // blocked or throttled: try API path or another provider after backoff
        return altProvider.Search(ctx, query, 10, false)
    }
    return nil, err
}

When it happens

Trigger: DuckDuckGo responds 202 (anomaly/bot challenge), 403, 429, or 5xx to the HTML search request issued via Search.

Common situations: Rate limiting from too many rapid scrapes; datacenter IPs flagged for bot challenges; missing/expired cookies or stale User-Agent; regional blocks.

Related errors

• bing API returned status %d: %s
• failed to create request: %w
• duckduckgo HTML search failed: %w
• failed to create request: %w
• failed to parse HTML: %w

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/88060cef22e4aff5. Report an issue: GitHub.