{"record":{"id":"1a07d498396bf9a2","repo":"Mintplex-Labs/anything-llm","slug":"empty-content-for-link-skipping","errorCode":null,"errorMessage":"Empty content for ${link}. Skipping.","messagePattern":"Empty content for (.+?)\\. Skipping\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"collector/utils/extensions/WebsiteDepth/index.js","lineNumber":171,"sourceCode":"    try {\n      const loader = new PuppeteerWebBaseLoader(link, {\n        launchOptions: {\n          headless: launchConfig.headless,\n          ignoreHTTPSErrors: true,\n          args: runtimeSettings.get(\"browserLaunchArgs\"),\n        },\n        gotoOptions: { waitUntil: \"networkidle2\" },\n        async evaluate(page, browser) {\n          const result = await page.evaluate(() => document.body.innerText);\n          await browser.close();\n          return result;\n        },\n      });\n      const docs = await loader.load();\n      const content = docs[0].pageContent;\n\n      if (!content.length) {\n        console.warn(`Empty content for ${link}. Skipping.`);\n        continue;\n      }\n\n      const url = new URL(link);\n      const decodedPathname = decodeURIComponent(url.pathname);\n      const filename = `${url.hostname}${decodedPathname.replace(/\\//g, \"_\")}`;\n\n      const data = {\n        id: v4(),\n        url: \"file://\" + slugify(filename) + \".html\",\n        title: slugify(filename) + \".html\",\n        docAuthor: \"no author found\",\n        description: \"No description found.\",\n        docSource: \"URL link uploaded by the user.\",\n        chunkSource: `link://${link}`,\n        published: new Date().toLocaleString(),\n        wordCount: content.split(\" \").length,\n        pageContent: content,","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/Mintplex-Labs/anything-llm/blob/20f6d3546c1938bfea1ad304f58a592dddcc5948/collector/utils/extensions/WebsiteDepth/index.js#L153-L189","documentation":"The WebsiteDepth crawler loaded the page with Puppeteer (waitUntil networkidle2) but document.body.innerText was an empty string, so the link is skipped and never embedded. The page 'loaded' yet produced no extractable text.","triggerScenarios":"SPA that renders content via XHR after networkidle fires; bot-detection serving an empty shell; a page that is all canvas/images/video; redirect chains landing on a blank interstitial; pages requiring interaction (cookie wall, login).","commonSituations":"React/Vue apps with late hydration; Cloudflare or Akamai challenges; links extracted from sitemaps that 200 but render nothing headless.","solutions":["Open the URL in a headless browser yourself to confirm what the DOM looks like without JS-rendered content.","Add a wait for a content-bearing selector or extra delay before extracting innerText.","If bot-blocking is the cause, run from an allowed IP/user-agent or source the content another way (API, raw HTML export).","Accept the skip — the crawler intentionally moves to the next link."],"exampleFix":"// before\ngotoOptions: { waitUntil: \"networkidle2\" },\n\n// after\ngotoOptions: { waitUntil: \"networkidle2\", timeout: 60000 },\n// and after load:\nawait page.waitForSelector(\"main, article, #content\", { timeout: 15000 }).catch(() => {});","handlingStrategy":"fallback","validationCode":"// Cheap preflight before deep-crawling a link:\nconst head = await fetch(link);\nconst type = head.headers.get('content-type') || '';\nif (!head.ok || !type.includes('text/html')) skipLink(link);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Add waitForSelector for a content container before extracting innerText.","Seed crawls with URLs known to render server-side or expose static HTML.","Run headless checks on target pages before scheduling deep crawls of them.","Treat skips as expected for bot-walled or fully client-rendered pages; source those via APIs instead."],"tags":["web-crawler","puppeteer","empty-content","website-depth"],"backgroundTag":"empty-page-content","analyzedSha":"20f6d3546c1938bfea1ad304f58a592dddcc5948","analyzedAt":"2026-08-18T10:02:21.017Z","contentChangedAt":"2026-08-18T10:02:21.017Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}