{"record":{"id":"37a5af13d598a16f","repo":"Tencent/WeKnora","slug":"failed-to-unmarshal-metaso-response-w","errorCode":null,"errorMessage":"failed to unmarshal Metaso response: %w","messagePattern":"failed to unmarshal Metaso response: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/infrastructure/web_search/metaso.go","lineNumber":117,"sourceCode":"\treq.Header.Set(\"Content-Type\", \"application/json\")\n\n\tlogger.Infof(ctx, \"[WebSearch][Metaso] query=%q maxResults=%d scope=%s\", query, maxResults, p.scope)\n\tresp, err := p.client.Do(req)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to execute Metaso request: %w\", err)\n\t}\n\tdefer resp.Body.Close()\n\trespBody, err := readMetasoResponseBody(resp.Body)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tif resp.StatusCode != http.StatusOK {\n\t\treturn nil, metasoHTTPError(resp.StatusCode, respBody)\n\t}\n\n\tvar response metasoSearchResponse\n\tif err := json.Unmarshal(respBody, &response); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to unmarshal Metaso response: %w\", err)\n\t}\n\tresults := make([]*types.WebSearchResult, 0, len(response.Webpages))\n\tfor _, item := range response.Webpages {\n\t\tif strings.TrimSpace(item.Title) == \"\" && strings.TrimSpace(item.Link) == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tsnippet := strings.TrimSpace(item.Summary)\n\t\tif snippet == \"\" {\n\t\t\tsnippet = strings.TrimSpace(item.Snippet)\n\t\t}\n\t\tresult := &types.WebSearchResult{Title: item.Title, URL: item.Link, Snippet: snippet, Content: item.RawContent, Source: \"metaso\"}\n\t\tif includeDate {\n\t\t\tif publishedAt, ok := parseMetasoDate(item.Date); ok {\n\t\t\t\tresult.PublishedAt = &publishedAt\n\t\t\t}\n\t\t}\n\t\tresults = append(results, result)\n\t\tif len(results) >= maxResults {","sourceCodeStart":99,"sourceCodeEnd":135,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/infrastructure/web_search/metaso.go#L99-L135","documentation":"Returned by MetasoProvider.Search when the HTTP response body was read successfully (status 200) but json.Unmarshal cannot decode it into metasoSearchResponse. This indicates the body is not the expected JSON shape — HTML from a proxy, truncated JSON, or a Metaso API schema change that no longer matches the struct. The json error is wrapped with %w for diagnosis.","triggerScenarios":"Search (internal/infrastructure/web_search/metaso.go:117): after a 200 status, json.Unmarshal(respBody, &response) fails — non-JSON HTML body, empty body, malformed/truncated JSON, or response fields with unexpected types vs metasoSearchResponse.","commonSituations":"TLS-intercepting proxy or captive portal injecting HTML with a 200 status; Metaso API contract change; response cut off by an aggressive timeout; custom gateway returning its own JSON envelope instead of Metaso's.","solutions":["Log the raw respBody on failure to see whether it is HTML, empty, or truncated JSON.","If the body is HTML, eliminate the intercepting proxy/captive portal or point the client directly at the Metaso endpoint.","Compare the actual JSON against metasoSearchResponse and update the struct if Metaso changed its schema.","Retry — a truncated body from a timeout may simply succeed on retry with a longer deadline."],"exampleFix":"// before: opaque failure\nif err := json.Unmarshal(respBody, &response); err != nil {\n    return nil, fmt.Errorf(\"failed to unmarshal Metaso response: %w\", err)\n}\n\n// after: surface body snippet for diagnosis\nif err := json.Unmarshal(respBody, &response); err != nil {\n    return nil, fmt.Errorf(\"failed to unmarshal Metaso response: %w (body: %.200s)\", err, string(respBody))\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func looksLikeMetasoResponse(b []byte) bool {\n    var probe struct {\n        Webpages []json.RawMessage `json:\"webpages\"`\n    }\n    return json.Unmarshal(b, &probe) == nil\n}\n\n// before full decode:\nif !looksLikeMetasoResponse(respBody) {\n    return nil, fmt.Errorf(\"unexpected Metaso response shape: %.200s\", string(respBody))\n}","tryCatchPattern":"if err := json.Unmarshal(respBody, &response); err != nil {\n    log.Printf(\"metaso unmarshal failed: %v, body=%.500s\", err, string(respBody))\n    if !utf8.Valid(respBody) || len(bytes.TrimSpace(respBody)) == 0 {\n        return nil, fmt.Errorf(\"empty/invalid body from metaso, retry advised: %w\", err)\n    }\n    return nil, fmt.Errorf(\"failed to unmarshal Metaso response: %w\", err)\n}","preventionTips":["Log a body snippet on every unmarshal failure to detect proxies injecting HTML.","Add integration tests decoding real captured Metaso responses to catch schema drift.","Pin or track the Metaso API version you depend on and review changelogs.","Extend client timeouts so responses are never truncated mid-body."],"tags":["json","unmarshal","schema","web-search"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}