What causes "Response was not valid JSON: ${url}" in jackwener/OpenCLI?

Log result.text on failure to inspect whether the body is HTML and what it says (challenge page, 404 page, etc.)

jackwener/OpenCLI · error · Error

Response was not valid JSON: ${url}

Error message

Response was not valid JSON: ${url}

What it means

Thrown by fetchJsonInBrowser when the HTTP response is ok but its body cannot be parsed as JSON. Uiverse's Remix _data endpoints normally return JSON, so this indicates the server returned HTML (an SPA fallback, error page, or bot-challenge page) instead of the expected loader data.

Source

Thrown at clis/uiverse/_shared.js:76

    const text = await response.text();
    return JSON.stringify({
      ok: response.ok,
      status: response.status,
      statusText: response.statusText,
      text,
      url,
    });
  })()`);

  const result = JSON.parse(raw);
  if (!result?.ok) {
    throw new Error(`Request failed: ${result?.status} ${result?.statusText} (${result?.url || url})`);
  }

  try {
    return JSON.parse(result.text);
  } catch {
    throw new Error(`Response was not valid JSON: ${url}`);
  }
}

export async function getPostDetails(page, input) {
  const normalized = parseComponentInput(input);
  await page.goto(normalized.url);

  const raw = await page.evaluate(`(async () => {
    const key = ${JSON.stringify(ROUTE_DATA_KEY)};
    const loaderData = window.__remixContext?.state?.loaderData || {};
    const routeData = loaderData[key];
    return JSON.stringify({ routeData: routeData || null, keys: Object.keys(loaderData) });
  })()`);

  const parsed = JSON.parse(raw);
  let routeData = parsed?.routeData;
  if (!routeData?.post?.id) {
    const routeUrl = `${normalized.url}?_data=${encodeURIComponent(ROUTE_DATA_KEY)}`;

View on GitHub (pinned to 49907e53dc)

Solutions

Log result.text on failure to inspect whether the body is HTML and what it says (challenge page, 404 page, etc.)
Check whether uiverse.io changed its Remix route data keys and update ROUTE_DATA_KEY/CODE_DATA_KEY in clis/uiverse/_shared.js
Retry later if the site is under maintenance or serving challenge pages; consider stealth-browser measures
Hit the endpoint manually in the logged-in browser (same URL) to confirm it still returns JSON

Example fix

// before
const payload = await fetchJsonInBrowser(page, codeUrl); // throws on HTML body
// after
let payload;
try {
  payload = await fetchJsonInBrowser(page, codeUrl);
} catch (e) {
  if (String(e.message).startsWith('Response was not valid JSON')) {
    const html = await page.content(); // inspect for challenge/route changes
    throw new Error(`Non-JSON response from ${codeUrl}; page title: ${await page.title()}`);
  }
  throw e;
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-check that the endpoint still returns JSON before scraping in bulk:
const probe = await page.evaluate(async (u) => {
  const r = await fetch(u, { credentials: 'include' });
  const t = await r.text();
  try { JSON.parse(t); return true; } catch { return false; }
}, probeUrl);
if (!probe) throw new Error('uiverse data endpoint no longer returns JSON; site may have changed');

Type guard

const looksLikeJson = (text) => /^[\[{]/.test(String(text).trim()) || /application\/json/i.test(String(text));

Try / catch

try {
  const data = await fetchJsonInBrowser(page, url);
} catch (e) {
  if (e.message.startsWith('Response was not valid JSON')) {
    const body = await page.content();
    console.error('Non-JSON body; challenge page?', body.slice(0, 500));
    throw new Error(`Site returned non-JSON for ${url}`);
  }
  throw e;
}

Prevention

Inspect non-JSON bodies to detect Cloudflare challenges vs. site redesigns
Pin/monitor ROUTE_DATA_KEY and CODE_DATA_KEY against live uiverse.io route names
Alert on this error in bulk jobs — it usually means a site-side change
Retry transiently, but never loop on this error without inspecting the body

When it happens

Trigger: The ?_data= route key no longer matches after a uiverse.io Remix route refactor, so the server returns the HTML page with 200; a Cloudflare/WAF interstitial HTML page served with 200; a login/redirect page returned with 200; an endpoint that changed shape and now returns plain text.

Common situations: Uiverse site upgrade changing route data keys ('routes/$username.$friendlyId' or 'routes/resource.post.code.$id'); scraping during a maintenance window returning HTML error pages; bot mitigation serving challenge HTML; VPN/proxy injecting HTML block pages.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/fcb6f3e989fa2040. Report an issue: GitHub.