{"record":{"id":"fcb6f3e989fa2040","repo":"jackwener/OpenCLI","slug":"response-was-not-valid-json-url","errorCode":null,"errorMessage":"Response was not valid JSON: ${url}","messagePattern":"Response was not valid JSON: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"clis/uiverse/_shared.js","lineNumber":76,"sourceCode":"    const text = await response.text();\n    return JSON.stringify({\n      ok: response.ok,\n      status: response.status,\n      statusText: response.statusText,\n      text,\n      url,\n    });\n  })()`);\n\n  const result = JSON.parse(raw);\n  if (!result?.ok) {\n    throw new Error(`Request failed: ${result?.status} ${result?.statusText} (${result?.url || url})`);\n  }\n\n  try {\n    return JSON.parse(result.text);\n  } catch {\n    throw new Error(`Response was not valid JSON: ${url}`);\n  }\n}\n\nexport async function getPostDetails(page, input) {\n  const normalized = parseComponentInput(input);\n  await page.goto(normalized.url);\n\n  const raw = await page.evaluate(`(async () => {\n    const key = ${JSON.stringify(ROUTE_DATA_KEY)};\n    const loaderData = window.__remixContext?.state?.loaderData || {};\n    const routeData = loaderData[key];\n    return JSON.stringify({ routeData: routeData || null, keys: Object.keys(loaderData) });\n  })()`);\n\n  const parsed = JSON.parse(raw);\n  let routeData = parsed?.routeData;\n  if (!routeData?.post?.id) {\n    const routeUrl = `${normalized.url}?_data=${encodeURIComponent(ROUTE_DATA_KEY)}`;","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/uiverse/_shared.js#L58-L94","documentation":"Thrown by fetchJsonInBrowser when the HTTP response is ok but its body cannot be parsed as JSON. Uiverse's Remix _data endpoints normally return JSON, so this indicates the server returned HTML (an SPA fallback, error page, or bot-challenge page) instead of the expected loader data.","triggerScenarios":"The ?_data= route key no longer matches after a uiverse.io Remix route refactor, so the server returns the HTML page with 200; a Cloudflare/WAF interstitial HTML page served with 200; a login/redirect page returned with 200; an endpoint that changed shape and now returns plain text.","commonSituations":"Uiverse site upgrade changing route data keys ('routes/$username.$friendlyId' or 'routes/resource.post.code.$id'); scraping during a maintenance window returning HTML error pages; bot mitigation serving challenge HTML; VPN/proxy injecting HTML block pages.","solutions":["Log result.text on failure to inspect whether the body is HTML and what it says (challenge page, 404 page, etc.)","Check whether uiverse.io changed its Remix route data keys and update ROUTE_DATA_KEY/CODE_DATA_KEY in clis/uiverse/_shared.js","Retry later if the site is under maintenance or serving challenge pages; consider stealth-browser measures","Hit the endpoint manually in the logged-in browser (same URL) to confirm it still returns JSON"],"exampleFix":"// before\nconst payload = await fetchJsonInBrowser(page, codeUrl); // throws on HTML body\n// after\nlet payload;\ntry {\n  payload = await fetchJsonInBrowser(page, codeUrl);\n} catch (e) {\n  if (String(e.message).startsWith('Response was not valid JSON')) {\n    const html = await page.content(); // inspect for challenge/route changes\n    throw new Error(`Non-JSON response from ${codeUrl}; page title: ${await page.title()}`);\n  }\n  throw e;\n}","handlingStrategy":"try-catch","validationCode":"// Pre-check that the endpoint still returns JSON before scraping in bulk:\nconst probe = await page.evaluate(async (u) => {\n  const r = await fetch(u, { credentials: 'include' });\n  const t = await r.text();\n  try { JSON.parse(t); return true; } catch { return false; }\n}, probeUrl);\nif (!probe) throw new Error('uiverse data endpoint no longer returns JSON; site may have changed');","typeGuard":"const looksLikeJson = (text) => /^[\\[{]/.test(String(text).trim()) || /application\\/json/i.test(String(text));","tryCatchPattern":"try {\n  const data = await fetchJsonInBrowser(page, url);\n} catch (e) {\n  if (e.message.startsWith('Response was not valid JSON')) {\n    const body = await page.content();\n    console.error('Non-JSON body; challenge page?', body.slice(0, 500));\n    throw new Error(`Site returned non-JSON for ${url}`);\n  }\n  throw e;\n}","preventionTips":["Inspect non-JSON bodies to detect Cloudflare challenges vs. site redesigns","Pin/monitor ROUTE_DATA_KEY and CODE_DATA_KEY against live uiverse.io route names","Alert on this error in bulk jobs — it usually means a site-side change","Retry transiently, but never loop on this error without inspecting the body"],"tags":["network","json","scraping","site-change"],"backgroundTag":"invalid-json-response","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}