{"record":{"id":"41a5b97947b882a1","repo":"Tencent/WeKnora","slug":"failed-to-unmarshal-response-w-41a5b9","errorCode":null,"errorMessage":"failed to unmarshal response: %w","messagePattern":"failed to unmarshal response: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/infrastructure/web_search/tavily.go","lineNumber":103,"sourceCode":"\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to execute request: %w\", err)\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode != http.StatusOK {\n\t\trespBody, _ := io.ReadAll(resp.Body)\n\t\tlogger.Warnf(ctx, \"[WebSearch][Tavily] API returned status %d: %s\", resp.StatusCode, string(respBody))\n\t\treturn nil, fmt.Errorf(\"tavily API returned status %d: %s\", resp.StatusCode, string(respBody))\n\t}\n\n\trespBody, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to read response: %w\", err)\n\t}\n\n\tvar respData tavilySearchResponse\n\tif err := json.Unmarshal(respBody, &respData); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to unmarshal response: %w\", err)\n\t}\n\n\tresults := make([]*types.WebSearchResult, 0, len(respData.Results))\n\tfor _, item := range respData.Results {\n\t\tresult := &types.WebSearchResult{\n\t\t\tTitle:   item.Title,\n\t\t\tURL:     item.URL,\n\t\t\tSnippet: item.Content,\n\t\t\tSource:  \"tavily\",\n\t\t}\n\t\tif includeDate && item.PublishedDate != \"\" {\n\t\t\tif t, err := time.Parse(time.RFC3339, item.PublishedDate); err == nil {\n\t\t\t\tresult.PublishedAt = &t\n\t\t\t}\n\t\t}\n\t\tresults = append(results, result)\n\t}\n\tlogger.Infof(ctx, \"[WebSearch][Tavily] returned %d results\", len(results))","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/infrastructure/web_search/tavily.go#L85-L121","documentation":"The Tavily API returned 200 but its body could not be parsed into tavilySearchResponse. This means the response is not the expected JSON structure — usually an auth HTML page, proxy interception page, or a schema change in the Tavily API.","triggerScenarios":"Calling provider.Search when the API returns HTML (e.g. a login/error page), an empty body, or JSON that no longer matches the tavilySearchResponse struct fields.","commonSituations":"Corporate proxy returning a captive-portal/interception page with 200 status, wrong base URL pointing at a non-API endpoint, Tavily introducing breaking API changes, or custom mock servers in tests returning malformed payloads.","solutions":["Log the raw respBody to see what was actually returned (HTML vs JSON).","Verify the base URL points to the official Tavily API endpoint (https://api.tavily.com/search).","Bypass any proxy that may intercept HTTPS responses.","Update the tavilySearchResponse struct if the Tavily API schema changed; add json tags matching the new payload."],"exampleFix":"// before\nvar respData tavilySearchResponse\nif err := json.Unmarshal(respBody, &respData); err != nil {\n    return nil, fmt.Errorf(\"failed to unmarshal response: %w\", err)\n}\n// after\nvar respData tavilySearchResponse\nif err := json.Unmarshal(respBody, &respData); err != nil {\n    logger.Warnf(ctx, \"unexpected tavily body: %s\", string(respBody))\n    return nil, fmt.Errorf(\"failed to unmarshal response: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":"// verify the endpoint returns JSON before wiring the provider\nresp, _ := http.Get(\"https://api.tavily.com\")\nct, _ := resp.Header.Get, \"\"\n_ = ct // Content-Type should be application/json on API responses","typeGuard":"func looksLikeJSON(b []byte) bool {\n    t := bytes.TrimSpace(b)\n    return len(t) > 0 && (t[0] == '{' || t[0] == '[')\n}","tryCatchPattern":"results, err := provider.Search(ctx, q, 5, false)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to unmarshal response\") {\n        // likely proxy interception or API schema change; log raw body and alert\n        return fmt.Errorf(\"tavily returned non-JSON payload; check proxy/baseURL: %w\", err)\n    }\n    return err\n}","preventionTips":["Confirm base URL points at the official Tavily API endpoint","Bypass HTTPS-intercepting proxies for API traffic","Pin/monitor the Tavily API version and test after provider updates","Log the raw response body on parse failure for diagnosis"],"tags":["json","parsing","web-search","tavily"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}