{"record":{"id":"0186bcc243d7ab7b","repo":"nexu-io/open-design","slug":"could-not-fetch-url-the-site-may-block-server","errorCode":null,"errorMessage":"Could not fetch ${url} — the site may block server-side requests.","messagePattern":"Could not fetch (.+?) — the site may block server-side requests\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"apps/daemon/src/brands/engine/build.ts","lineNumber":420,"sourceCode":"      radius: `${seed.borderRadius}px`,\n      borderWeight: `${seed.lineWidth}px`,\n      spacing: \"8px baseline grid\",\n      postureRules: [],\n    },\n  };\n}\n\n/**\n * Build a BrandSystem straight from a site URL — the deterministic, no-LLM\n * path. `prefetchBrand` must write to a brand dir, so we hand it a throwaway\n * temp dir, read back any downloaded logos into the bundle, then assemble.\n */\nexport async function buildFromUrl(url: string, opts?: { slug?: string }): Promise<BrandSystem> {\n  const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), \"brand-prefetch-\"));\n  try {\n    const material = await prefetchBrand(url, tmpDir);\n    if (!material) {\n      throw new Error(`Could not fetch ${url} — the site may block server-side requests.`);\n    }\n\n    const seed = seedFromMaterial(material);\n    const brand = brandFromMaterial(material, seed);\n    const slug = opts?.slug ? slugify(opts.slug) : slugify(brand.name);\n\n    // Pull any downloaded logos back into the in-memory bundle.\n    const extraFiles: Record<string, string> = {};\n    for (const logo of material.logos ?? []) {\n      const abs = path.join(tmpDir, \"logos\", logo.file);\n      try {\n        if (logo.contentType?.includes(\"svg\") || logo.file.endsWith(\".svg\")) {\n          extraFiles[`logos/${logo.file}`] = fs.readFileSync(abs, \"utf8\");\n        } else {\n          // Binary assets: keep as base64 data so the bundle stays string-only.\n          const b64 = fs.readFileSync(abs).toString(\"base64\");\n          extraFiles[`logos/${logo.file}.b64`] = b64;\n        }","sourceCodeStart":402,"sourceCodeEnd":438,"githubUrl":"https://github.com/nexu-io/open-design/blob/5be4028344c2eb4c667c5a97bda8f750c5597ef7/apps/daemon/src/brands/engine/build.ts#L402-L438","documentation":"Thrown by buildFromUrl in the deterministic, no-LLM brand builder. prefetchBrand(url, tmpDir) returned a falsy material, meaning the site could not be fetched, returned no usable logos/content, or the request failed upstream. This path is offline and deterministic, so it does not fall back to an agent.","triggerScenarios":"The target site blocks server-side scrapers (403/429), requires JavaScript to render, has no logos or readable content, returns an empty body, the daemon host cannot reach it (DNS/firewall), or the TLS cert is invalid.","commonSituations":"Cloudflare/Akamai-protected marketing sites, JS-only SPAs with no SSR HTML, intranet URLs unreachable from the daemon, geo-blocked domains, sites that need cookies/consent.","solutions":["Switch to the agent-driven extraction path (startBrandExtraction) which is more tolerant of JS-heavy and protected sites.","From the daemon host, curl the URL to confirm reachability and inspect what the server returns.","If you control the site, allowlist the daemon's egress IP or set a recognizable User-Agent.","Fall back to pasting a DESIGN.md (startBrandExtraction accepts designMd instead of url)."],"exampleFix":"// before: deterministic builder cannot scrape a JS-only site\nconst brand = await buildFromUrl('https://app.example.com');\n\n// after: use the agent-driven path, or supply design markdown\nconst result = await startBrandExtraction({ url: 'https://app.example.com', /* deps */ });\n// or\nconst result = await startBrandExtraction({ designMd: '<paste DESIGN.md>', description: '...', /* deps */ });","handlingStrategy":"fallback","validationCode":"// Probe reachability before invoking the deterministic builder.\nasync function isFetchable(url: string): Promise<boolean> {\n  try {\n    const res = await fetch(url, { redirect: 'follow', headers: { 'user-agent': 'OpenDesign-Brand-Build/1.0' } });\n    return res.ok && (await res.text()).length > 0;\n  } catch {\n    return false;\n  }\n}\nif (!await isFetchable(url)) {\n  // skip deterministic build, go straight to agent-driven extraction\n}","typeGuard":null,"tryCatchPattern":"let brand;\ntry {\n  brand = await buildFromUrl(url);\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith('Could not fetch ')) {\n    // fall back to the agent-driven path which tolerates JS/protected sites\n    brand = await startBrandExtraction({ url, /* deps */ }).then(r => r.brand);\n  } else {\n    throw err;\n  }\n}","preventionTips":["Prefer startBrandExtraction (agent-driven) for JS-heavy or scraper-protected sites.","Verify egress from the daemon host with curl before reporting a site as broken.","Keep a DESIGN.md fallback so brand creation does not depend on live scraping."],"tags":["brands","network","scraping","deterministic-build"],"backgroundTag":null,"analyzedSha":"5be4028344c2eb4c667c5a97bda8f750c5597ef7","analyzedAt":"2026-08-12T12:03:58.812Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}