{"record":{"id":"d85625c9b575a80a","repo":"jackwener/OpenCLI","slug":"batch-fetch-failed-for-urls-i-r-as-error","errorCode":null,"errorMessage":"Batch fetch failed for ${urls[i]}: ${(r as { error: string }).error}","messagePattern":"Batch fetch failed for (.+?): (.+?)\\)\\.error\\}","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/pipeline/steps/fetch.ts","lineNumber":120,"sourceCode":"    const renderedParams: Record<string, string> = {};\n    for (const [k, v] of Object.entries(queryParams)) renderedParams[k] = String(render(v, { args, data }));\n\n    const urls = data.map((item, index) => {\n      let url = String(render(urlTemplate, { args, data, item, index }));\n      if (Object.keys(renderedParams).length > 0) {\n        const qs = new URLSearchParams(renderedParams).toString();\n        url = `${url}${url.includes('?') ? '&' : '?'}${qs}`;\n      }\n      return url;\n    });\n\n    // BATCH IPC: if browser is available, batch all fetches into a single evaluate() call\n    if (page !== null) {\n      const results = await fetchBatchInBrowser(page, urls, method.toUpperCase(), renderedHeaders, concurrency);\n      for (let i = 0; i < results.length; i++) {\n        const r = results[i];\n        if (r && typeof r === 'object' && 'error' in r) {\n          log.warn(`Batch fetch failed for ${urls[i]}: ${(r as { error: string }).error}`);\n        }\n      }\n      return results;\n    }\n\n    // Non-browser: use concurrent pool (already optimized)\n    return mapConcurrent(data, concurrency, async (item, index) => {\n      const itemUrl = String(render(urlTemplate, { args, data, item, index }));\n      try {\n        return await fetchSingle(null, itemUrl, method, queryParams, headers, args, data);\n      } catch (error) {\n        const message = getErrorMessage(error);\n        log.warn(`Batch fetch failed for ${itemUrl}: ${message}`);\n        return { error: message };\n      }\n    });\n  }\n  const url = render(urlOrObj, { args, data });","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/src/pipeline/steps/fetch.ts#L102-L138","documentation":"In stepFetch's browser path, all URLs are fetched in one batched page.evaluate() call via fetchBatchInBrowser. Each result object may carry an error field set inside the browser context (network failure, HTTP error, navigation abort). The step logs this warning per failed URL and still returns the results array, so the error is per-item data, not a thrown exception.","triggerScenarios":"fetchBatchInBrowser returns { error: string } for a given URL — DNS resolution failure in the browser page, connection refused/timeout, aborted navigation, or a JavaScript exception inside the in-page fetch wrapper for that item.","commonSituations":"Target site blocks headless browsers (403/challenge); invalid or expired URL from template rendering; corporate proxy/firewall; one bad URL among many in a batch from scraped data.","solutions":["Check the error string per URL and re-fetch failed items individually or with retries/backoff.","Validate rendered URLs (scheme, host, encoding) before batching — template artifacts often produce malformed URLs.","Set realistic timeouts/headers (User-Agent) since sites commonly reject headless defaults.","Filter out known-bad data rows upstream so one bad entry doesn't pollute the batch."],"exampleFix":"// before\nconst results = await fetchBatchInBrowser(page, urls, ...);\n// after\nconst results = await fetchBatchInBrowser(page, urls, ...);\nconst failed = urls.filter((_, i) => results[i]?.error);\nfor (const u of failed) log.warn(`retrying ${u}`);\n// retry failed items with a sequential fetchSingle pass","handlingStrategy":"retry","validationCode":"const validUrls = urls.filter((u) => { try { new URL(u); return true; } catch { return false; } });","typeGuard":"function hasFetchError(r: unknown): r is { error: string } {\n  return typeof r === 'object' && r !== null && 'error' in r && typeof (r as { error: unknown }).error === 'string';\n}","tryCatchPattern":"const results = await fetchBatchInBrowser(page, urls, method, headers, concurrency);\nfor (let i = 0; i < results.length; i++) {\n  if (hasFetchError(results[i])) {\n    await withRetry(() => fetchSingle(null, urls[i], ...), { retries: 3, backoffMs: 500 });\n  }\n}","preventionTips":["Validate every rendered URL with new URL() before batching.","Set explicit timeouts and realistic User-Agent headers.","Add per-item retry with backoff for transient network errors.","Sanitize upstream data so bad rows don't yield malformed URLs."],"tags":["network","fetch","browser"],"backgroundTag":"fetch-request-failed","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}