Tencent/WeKnora · error

failed to unmarshal response: %w

Error message

failed to unmarshal response: %w

What it means

The Tavily API returned 200 but its body could not be parsed into tavilySearchResponse. This means the response is not the expected JSON structure — usually an auth HTML page, proxy interception page, or a schema change in the Tavily API.

Source

Thrown at internal/infrastructure/web_search/tavily.go:103

	if err != nil {
		return nil, fmt.Errorf("failed to execute request: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		respBody, _ := io.ReadAll(resp.Body)
		logger.Warnf(ctx, "[WebSearch][Tavily] API returned status %d: %s", resp.StatusCode, string(respBody))
		return nil, fmt.Errorf("tavily API returned status %d: %s", resp.StatusCode, string(respBody))
	}

	respBody, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("failed to read response: %w", err)
	}

	var respData tavilySearchResponse
	if err := json.Unmarshal(respBody, &respData); err != nil {
		return nil, fmt.Errorf("failed to unmarshal response: %w", err)
	}

	results := make([]*types.WebSearchResult, 0, len(respData.Results))
	for _, item := range respData.Results {
		result := &types.WebSearchResult{
			Title:   item.Title,
			URL:     item.URL,
			Snippet: item.Content,
			Source:  "tavily",
		}
		if includeDate && item.PublishedDate != "" {
			if t, err := time.Parse(time.RFC3339, item.PublishedDate); err == nil {
				result.PublishedAt = &t
			}
		}
		results = append(results, result)
	}
	logger.Infof(ctx, "[WebSearch][Tavily] returned %d results", len(results))

View on GitHub (pinned to 988cbb0330)

Solutions

1. Log the raw respBody to see what was actually returned (HTML vs JSON).
2. Verify the base URL points to the official Tavily API endpoint (https://api.tavily.com/search).
3. Bypass any proxy that may intercept HTTPS responses.
4. Update the tavilySearchResponse struct if the Tavily API schema changed; add json tags matching the new payload.

Example fix

// before
var respData tavilySearchResponse
if err := json.Unmarshal(respBody, &respData); err != nil {
    return nil, fmt.Errorf("failed to unmarshal response: %w", err)
}
// after
var respData tavilySearchResponse
if err := json.Unmarshal(respBody, &respData); err != nil {
    logger.Warnf(ctx, "unexpected tavily body: %s", string(respBody))
    return nil, fmt.Errorf("failed to unmarshal response: %w", err)
}

// verify the endpoint returns JSON before wiring the provider
resp, _ := http.Get("https://api.tavily.com")
ct, _ := resp.Header.Get, ""
_ = ct // Content-Type should be application/json on API responses

func looksLikeJSON(b []byte) bool {
    t := bytes.TrimSpace(b)
    return len(t) > 0 && (t[0] == '{' || t[0] == '[')
}

results, err := provider.Search(ctx, q, 5, false)
if err != nil {
    if strings.Contains(err.Error(), "failed to unmarshal response") {
        // likely proxy interception or API schema change; log raw body and alert
        return fmt.Errorf("tavily returned non-JSON payload; check proxy/baseURL: %w", err)
    }
    return err
}

When it happens

Trigger: Calling provider.Search when the API returns HTML (e.g. a login/error page), an empty body, or JSON that no longer matches the tavilySearchResponse struct fields.

Common situations: Corporate proxy returning a captive-portal/interception page with 200 status, wrong base URL pointing at a non-API endpoint, Tavily introducing breaking API changes, or custom mock servers in tests returning malformed payloads.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• failed to unmarshal response: %w
• decode suggestion JSON: %w
• failed to parse HTML: %w
• failed to unmarshal Exa response: %w
• failed to marshal request: %w

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/41a5b97947b882a1. Report an issue: GitHub.