{"record":{"id":"b0bc3cfcd42ad96a","repo":"jackwener/OpenCLI","slug":"timed-out-waiting-for-network-idle-after-waitsec","errorCode":null,"errorMessage":"Timed out waiting for network idle after ${waitSeconds}s","messagePattern":"Timed out waiting for network idle after (.+?)s","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"clis/web/read.js","lineNumber":443,"sourceCode":"        await page.goto(url);\n        if (kwargs['wait-for']) {\n            const waitResult = await page.evaluate(buildWaitForSelectorAcrossFramesJs(String(kwargs['wait-for']), waitSeconds * 1000));\n            if (waitResult?.invalidSelector) {\n                throw new Error(`Invalid --wait-for selector \"${kwargs['wait-for']}\": ${waitResult.error || 'querySelector failed'}`);\n            }\n            if (!waitResult?.ok) {\n                throw new Error(`Timed out waiting for selector \"${kwargs['wait-for']}\" in main document or same-origin iframes`);\n            }\n        } else if (waitUntil !== 'networkidle') {\n            await page.wait(waitSeconds);\n        }\n        if (waitUntil === 'networkidle') {\n            if (!captureSupported) {\n                throw new Error('Network capture is unavailable, so --wait-until networkidle cannot be satisfied');\n            }\n            const idle = await waitForNetworkIdle(page, waitSeconds, networkEntries);\n            if (!idle?.ok) {\n                throw new Error(`Timed out waiting for network idle after ${waitSeconds}s`);\n            }\n        }\n        // Extract article content using browser-side heuristics\n        const data = await page.evaluate(buildRenderAwareExtractorJs({ frames: frameMode }));\n        if (captureSupported) await drainNetworkCapture(page, networkEntries);\n        if (shouldDiagnose) process.stderr.write(formatDiagnostics(data, networkEntries, captureSupported));\n        // Determine Referer from URL for image downloads\n        let referer = '';\n        try {\n            const parsed = new URL(url);\n            referer = parsed.origin + '/';\n        }\n        catch { /* ignore */ }\n        const result = await downloadArticle({\n            title: data?.title || 'untitled',\n            author: data?.author,\n            publishTime: data?.publishTime,\n            sourceUrl: url,","sourceCodeStart":425,"sourceCodeEnd":461,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/web/read.js#L425-L461","documentation":"With --wait-until networkidle, the command waits (waitForNetworkIdle) for the page's network activity to settle within the --wait window. If requests keep firing (or never finish) when the budget expires, waitForNetworkIdle returns not-ok and this timeout error is thrown.","triggerScenarios":"Pages with long-polling/WebSocket/streaming requests, analytics beacons that keep firing, slow or hanging third-party scripts, or a very small --wait value — any case where in-flight/ongoing requests persist past waitSeconds so the idle check returns ok:false.","commonSituations":"Scraping dashboards with auto-refresh polling; pages with chat/telemetry sockets; ad-heavy pages with slow trackers; running on slow networks with the default short timeout.","solutions":["Increase the timeout: --wait 30 (or higher) to give the idle detector more budget.","Abandon networkidle for this page and use --wait-for <selector> on a stable element.","Check devtools network tab for endless polling/streaming endpoints; if present, networkidle is unattainable by design.","Block noisy trackers/scripts (via proxy/adblock or the tool's blocking options) so activity can settle.","Verify the page isn't stuck retrying failed requests due to auth or network issues."],"exampleFix":"// before\nweb read https://dashboard.example.com --wait-until networkidle --wait 3\n// after\nweb read https://dashboard.example.com --wait-until networkidle --wait 30\n// or, if the page polls forever:\n// web read https://dashboard.example.com --wait-for '#dashboard-ready'","handlingStrategy":"retry","validationCode":"// Detect endless-polling pages before choosing networkidle\nconst hasPolling = await page.evaluate(`performance.getEntriesByType('resource')\n  .filter(r => r.initiatorType === 'xmlhttprequest' || r.initiatorType === 'fetch').length > 20`);\nif (hasPolling) useWaitForSelectorInstead = true;","typeGuard":"function isIdleResult(r) { return r != null && typeof r === 'object' && typeof r.ok === 'boolean'; }","tryCatchPattern":"try {\n  await readPage(url, { waitUntil: 'networkidle', waitSeconds: 30 });\n} catch (e) {\n  if (/Timed out waiting for network idle/.test(e.message)) {\n    await readPage(url, { waitFor: '#content', waitSeconds: 30 }); // deterministic fallback\n  } else throw e;\n}","preventionTips":["Never use networkidle on pages with long-polling, SSE, or WebSockets","Give networkidle a generous --wait budget (20s+)","Prefer --wait-for element readiness for deterministic pages","Block trackers/ads in the automation profile to let the network settle"],"tags":["browser","network","timeout","scraping"],"backgroundTag":"network-idle-timeout","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}