{"record":{"id":"1d935b227c3daf92","repo":"Tencent/WeKnora","slug":"readability-parse-w","errorCode":null,"errorMessage":"readability parse: %w","messagePattern":"readability parse: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"internal/datasource/connector/rss/client.go","lineNumber":112,"sourceCode":"// fetchFeed retrieves the raw bytes of a feed document.\nfunc (c *client) fetchFeed(ctx context.Context, feedURL string) ([]byte, error) {\n\treturn c.fetch(ctx, feedURL, maxFeedSize, true)\n}\n\n// extractArticle fetches an article page and returns the readability-cleaned\n// main content as HTML, plus the extracted title (may be empty). Returns an\n// error if the page can't be fetched or no readable content is found, so the\n// caller can fall back to feed-provided content.\nfunc (c *client) extractArticle(ctx context.Context, articleURL string) (contentHTML, title string, err error) {\n\tbody, err := c.fetch(ctx, articleURL, maxArticleSize, false)\n\tif err != nil {\n\t\treturn \"\", \"\", err\n\t}\n\n\tpageURL, _ := url.Parse(articleURL)\n\tarticle, err := readability.FromReader(bytes.NewReader(body), pageURL)\n\tif err != nil {\n\t\treturn \"\", \"\", fmt.Errorf(\"readability parse: %w\", err)\n\t}\n\tif article.Node == nil {\n\t\treturn \"\", \"\", fmt.Errorf(\"no readable content extracted\")\n\t}\n\n\tvar buf bytes.Buffer\n\tif err := article.RenderHTML(&buf); err != nil {\n\t\treturn \"\", \"\", fmt.Errorf(\"render article html: %w\", err)\n\t}\n\treturn buf.String(), article.Title(), nil\n}\n","sourceCodeStart":94,"sourceCodeEnd":124,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/datasource/connector/rss/client.go#L94-L124","documentation":"Returned by extractArticle when go-readability cannot parse the fetched article page into a DOM. readability.FromReader parses the HTML with golang-net-html and runs the readability scoring algorithm; a parse error means the bytes are not valid HTML/XML at all (e.g. binary data, a JSON error page, or corrupt encoding), not merely 'no article found' (that is error 715).","triggerScenarios":"The article URL returned non-HTML content (a PDF, image, JSON API error, or plain text); the page uses an encoding the parser mangles into invalid markup; or the fetch returned an HTML error page so broken that the HTML parser itself fails.","commonSituations":"Feed item links point to PDFs or image attachments; sites behind bot protection serving binary challenge pages; legacy feeds with misdeclared charsets (e.g. claiming UTF-8 while serving GBK); article URL pointing at a JSON REST endpoint.","solutions":["Check what the URL actually returns (curl -i) — if it is not HTML (PDF/image/JSON), exclude that item or handle non-HTML link types before calling extractArticle.","Fix charset declarations on the source site or transcode the body to UTF-8 before parsing.","If a bot-protection page is the cause, the content cannot be extracted by design; fall back to feed-provided summary content, as the caller resolveItem does.","Verify the fetch got a real article page, not an error page, by sniffing Content-Type before parsing."],"exampleFix":"// before\ncontentHTML, title, err := cli.extractArticle(ctx, item.Link)\nif err != nil {\n    contentHTML = item.Content\n}\n// after\nct := respHeader.Get(\"Content-Type\")\nif strings.Contains(ct, \"text/html\") {\n    contentHTML, title, err = cli.extractArticle(ctx, item.Link)\n}\nif err != nil || contentHTML == \"\" {\n    contentHTML = item.Content // feed-provided fallback\n}","handlingStrategy":"fallback","validationCode":"func isHTMLContent(resp *http.Response) bool {\n    ct := resp.Header.Get(\"Content-Type\")\n    return strings.Contains(ct, \"text/html\") || strings.Contains(ct, \"application/xhtml\")\n}","typeGuard":null,"tryCatchPattern":"contentHTML, title, err := cli.extractArticle(ctx, item.Link)\nif err != nil {\n    log.Warnf(\"extract %s failed: %v; falling back to feed content\", item.Link, err)\n    contentHTML, title = item.Content, item.Title\n}","preventionTips":["Check Content-Type before running readability and skip non-HTML links.","Prefer feed-provided item.Content/Description as an always-available fallback.","Handle charsets explicitly: transcode to UTF-8 before parsing.","Exclude PDF/image link patterns from article extraction."],"tags":["html","parsing","rss","readability"],"backgroundTag":"html-parse-failed","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}