{"record":{"id":"63368ffd57c88fad","repo":"Tencent/WeKnora","slug":"no-readable-content-extracted","errorCode":null,"errorMessage":"no readable content extracted","messagePattern":"no readable content extracted","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"internal/datasource/connector/rss/client.go","lineNumber":115,"sourceCode":"}\n\n// extractArticle fetches an article page and returns the readability-cleaned\n// main content as HTML, plus the extracted title (may be empty). Returns an\n// error if the page can't be fetched or no readable content is found, so the\n// caller can fall back to feed-provided content.\nfunc (c *client) extractArticle(ctx context.Context, articleURL string) (contentHTML, title string, err error) {\n\tbody, err := c.fetch(ctx, articleURL, maxArticleSize, false)\n\tif err != nil {\n\t\treturn \"\", \"\", err\n\t}\n\n\tpageURL, _ := url.Parse(articleURL)\n\tarticle, err := readability.FromReader(bytes.NewReader(body), pageURL)\n\tif err != nil {\n\t\treturn \"\", \"\", fmt.Errorf(\"readability parse: %w\", err)\n\t}\n\tif article.Node == nil {\n\t\treturn \"\", \"\", fmt.Errorf(\"no readable content extracted\")\n\t}\n\n\tvar buf bytes.Buffer\n\tif err := article.RenderHTML(&buf); err != nil {\n\t\treturn \"\", \"\", fmt.Errorf(\"render article html: %w\", err)\n\t}\n\treturn buf.String(), article.Title(), nil\n}\n","sourceCodeStart":97,"sourceCodeEnd":124,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/datasource/connector/rss/client.go#L97-L124","documentation":"Returned when readability successfully parsed the HTML but its scoring algorithm found no readable article node (article.Node == nil). The readability heuristic requires a minimum amount of text-bearing, structured content; pages that are too small, JS-shell single-page apps, or pure link lists yield no candidate article.","triggerScenarios":"The article page is a JavaScript-rendered SPA whose HTML contains no server-rendered text; the page has less content than readability's scoring thresholds; the URL redirected to a homepage, login page, or paywall stub; the page is a bare link aggregator with no body text.","commonSituations":"Modern news sites serving empty shells to non-browser clients; paywalled articles returning a stub page; feeds whose item links point to landing pages instead of articles; very short link-list posts (e.g. linkblogs).","solutions":["Fall back to the feed-provided content (item.Content / item.Description) — resolveItem is designed to do exactly this; ensure that fallback path is in place.","For JS-only sites, no static fetch will work; either exclude the item or use feed-provided content.","Detect login/paywall stubs (small body size, keywords like 'subscribe') and skip extraction for those domains.","Consider a rendering proxy (headless browser service) if full text from SPAs is a hard requirement."],"exampleFix":"// before\ncontentHTML, _, err := cli.extractArticle(ctx, item.Link)\nif err != nil {\n    return item.Summary\n}\n// after\ncontentHTML, _, err := cli.extractArticle(ctx, item.Link)\nif err != nil || utf8.RuneCountInString(contentHTML) < 200 {\n    return coalesce(item.Content, item.Description, item.Title)\n}","handlingStrategy":"fallback","validationCode":"func looksLikeArticle(html string) bool {\n    doc, err := goquery.NewDocumentFromReader(strings.NewReader(html))\n    if err != nil {\n        return false\n    }\n    text := strings.TrimSpace(doc.Find(\"body\").Text())\n    return utf8.RuneCountInString(text) > 200\n}","typeGuard":null,"tryCatchPattern":"contentHTML, title, err := cli.extractArticle(ctx, item.Link)\nif err != nil || contentHTML == \"\" {\n    // SPA/paywall/landing page — no readable node\n    contentHTML = coalesce(item.Content, item.Description)\n    title = item.Title\n}","preventionTips":["Always keep the feed-provided content fallback path; never treat full-text extraction as guaranteed.","Detect paywall/login stubs by size and keywords and skip extraction for those domains.","Maintain a per-domain skip list for known JS-only sites.","Accept shorter articles rather than treating small extractions as failures."],"tags":["readability","html","rss","content-extraction"],"backgroundTag":"no-readable-content","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}