What causes "Batch fetch failed for ${urls[i]}: ${(r as { error: string }).error}" in jackwener/OpenCLI?

Check the error string per URL and re-fetch failed items individually or with retries/backoff.

Batch fetch failed for ${urls[i]}: ${(r as { error: string }

Error message

Batch fetch failed for ${urls[i]}: ${(r as { error: string }).error}

What it means

In stepFetch's browser path, all URLs are fetched in one batched page.evaluate() call via fetchBatchInBrowser. Each result object may carry an error field set inside the browser context (network failure, HTTP error, navigation abort). The step logs this warning per failed URL and still returns the results array, so the error is per-item data, not a thrown exception.

Source

Thrown at src/pipeline/steps/fetch.ts:120

    const renderedParams: Record<string, string> = {};
    for (const [k, v] of Object.entries(queryParams)) renderedParams[k] = String(render(v, { args, data }));

    const urls = data.map((item, index) => {
      let url = String(render(urlTemplate, { args, data, item, index }));
      if (Object.keys(renderedParams).length > 0) {
        const qs = new URLSearchParams(renderedParams).toString();
        url = `${url}${url.includes('?') ? '&' : '?'}${qs}`;
      }
      return url;
    });

    // BATCH IPC: if browser is available, batch all fetches into a single evaluate() call
    if (page !== null) {
      const results = await fetchBatchInBrowser(page, urls, method.toUpperCase(), renderedHeaders, concurrency);
      for (let i = 0; i < results.length; i++) {
        const r = results[i];
        if (r && typeof r === 'object' && 'error' in r) {
          log.warn(`Batch fetch failed for ${urls[i]}: ${(r as { error: string }).error}`);
        }
      }
      return results;
    }

    // Non-browser: use concurrent pool (already optimized)
    return mapConcurrent(data, concurrency, async (item, index) => {
      const itemUrl = String(render(urlTemplate, { args, data, item, index }));
      try {
        return await fetchSingle(null, itemUrl, method, queryParams, headers, args, data);
      } catch (error) {
        const message = getErrorMessage(error);
        log.warn(`Batch fetch failed for ${itemUrl}: ${message}`);
        return { error: message };
      }
    });
  }
  const url = render(urlOrObj, { args, data });

View on GitHub (pinned to 49907e53dc)

Solutions

Check the error string per URL and re-fetch failed items individually or with retries/backoff.
Validate rendered URLs (scheme, host, encoding) before batching — template artifacts often produce malformed URLs.
Set realistic timeouts/headers (User-Agent) since sites commonly reject headless defaults.
Filter out known-bad data rows upstream so one bad entry doesn't pollute the batch.

Example fix

// before
const results = await fetchBatchInBrowser(page, urls, ...);
// after
const results = await fetchBatchInBrowser(page, urls, ...);
const failed = urls.filter((_, i) => results[i]?.error);
for (const u of failed) log.warn(`retrying ${u}`);
// retry failed items with a sequential fetchSingle pass

Defensive patterns

Strategy: retry

Validate before calling

const validUrls = urls.filter((u) => { try { new URL(u); return true; } catch { return false; } });

Type guard

function hasFetchError(r: unknown): r is { error: string } {
  return typeof r === 'object' && r !== null && 'error' in r && typeof (r as { error: unknown }).error === 'string';
}

Try / catch

const results = await fetchBatchInBrowser(page, urls, method, headers, concurrency);
for (let i = 0; i < results.length; i++) {
  if (hasFetchError(results[i])) {
    await withRetry(() => fetchSingle(null, urls[i], ...), { retries: 3, backoffMs: 500 });
  }
}

Prevention

Validate every rendered URL with new URL() before batching.
Set explicit timeouts and realistic User-Agent headers.
Add per-item retry with backoff for transient network errors.
Sanitize upstream data so bad rows don't yield malformed URLs.

When it happens

Trigger: fetchBatchInBrowser returns { error: string } for a given URL — DNS resolution failure in the browser page, connection refused/timeout, aborted navigation, or a JavaScript exception inside the in-page fetch wrapper for that item.

Common situations: Target site blocks headless browsers (403/challenge); invalid or expired URL from template rendering; corporate proxy/firewall; one bad URL among many in a batch from scraped data.

Related errors

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/d85625c9b575a80a. Report an issue: GitHub.