{"record":{"id":"5873b07e3d2686bf","repo":"lissy93/web-check","slug":"crt-sh-returned-an-unexpected-response","errorCode":null,"errorMessage":"crt.sh returned an unexpected response","messagePattern":"crt\\.sh returned an unexpected response","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"api/subdomains.js","lineNumber":30,"sourceCode":"\nconst certSpotter = async (domain) => {\n  const token = process.env.CERTSPOTTER_TOKEN;\n  const res = await httpGet('https://api.certspotter.com/v1/issuances', {\n    params: { domain, include_subdomains: 'true', expand: 'dns_names' },\n    headers: { Accept: 'application/json', ...(token && { Authorization: `Bearer ${token}` }) },\n    timeout: SOURCE_TIMEOUT,\n  });\n  if (!Array.isArray(res.data)) throw new Error('certSpotter returned an unexpected response');\n  return res.data.flatMap((row) => (Array.isArray(row?.dns_names) ? row.dns_names : []));\n};\n\nconst crtSh = async (domain) => {\n  const res = await httpGet('https://crt.sh/', {\n    params: { q: `%.${domain}`, output: 'json' },\n    headers: { Accept: 'application/json' },\n    timeout: SOURCE_TIMEOUT,\n  });\n  if (!Array.isArray(res.data)) throw new Error('crt.sh returned an unexpected response');\n  return res.data.flatMap((row) => String(row?.name_value ?? '').split('\\n'));\n};\n\nconst hackerTarget = async (domain) => {\n  const res = await httpGet('https://api.hackertarget.com/hostsearch/', {\n    params: { q: domain },\n    timeout: SOURCE_TIMEOUT,\n  });\n  const body = typeof res.data === 'string' ? res.data : '';\n  if (!body || /error|api count|quota/i.test(body)) throw new Error('hackerTarget unavailable');\n  return body.split('\\n').map((line) => line.split(',')[0]);\n};\n\nconst SOURCES = [\n  { name: 'certSpotter', lookup: certSpotter },\n  { name: 'crt.sh', lookup: crtSh },\n  { name: 'hackerTarget', lookup: hackerTarget },\n];","sourceCodeStart":12,"sourceCodeEnd":48,"githubUrl":"https://github.com/lissy93/web-check/blob/af1a97759fc8bcc43c876c94f2ccb018ce215f90/api/subdomains.js#L12-L48","documentation":"crtSh fetches crt.sh's JSON issuance feed and expects an array; a non-array body (HTML error page, rate-limit text, empty body on crt.sh's frequent 503s) triggers this throw. crt.sh is a free, unauthenticated service with known instability under load.","triggerScenarios":"crt.sh returns its common 'Server Error'/503 HTML page, rate-limits the caller, or returns a JSON error object instead of the issuance array.","commonSituations":"Bulk subdomain scans hitting crt.sh too fast, crt.sh being intermittently down (well-known), or timeout truncation producing a partial/empty body.","solutions":["Treat crt.sh as best-effort: catch per-source and continue with other SOURCES (aggregation already exists)","Throttle/retry crt.sh with exponential backoff (it is heavily rate-limited)","If persistent, temporarily remove crt.sh from SOURCES until it recovers"],"exampleFix":"// before\nconst found = await crtSh(domain); // single-source dependency\n\n// after\nconst safe = async (fn, name) => { try { return await fn(domain); } catch (e) { console.warn(`${name} unavailable: ${e.message}`); return []; } };\nconst found = [...await safe(certSpotter, 'certSpotter'), ...await safe(crtSh, 'crt.sh'), ...await safe(hackerTarget, 'hackerTarget')];","handlingStrategy":"retry","validationCode":"null","typeGuard":"null","tryCatchPattern":"const withRetry = async (fn, tries = 3) => {\n  for (let i = 0; i < tries; i++) {\n    try { return await fn(domain); }\n    catch (e) { if (i === tries - 1) { console.warn(`crt.sh failed: ${e.message}`); return []; } await new Promise(r => setTimeout(r, 2 ** i * 1000)); }\n  }\n};","preventionTips":["Cache crt.sh results per domain to avoid repeat calls","Throttle concurrent scans; crt.sh rate-limits aggressively","Run multiple SOURCES concurrently with allSettled so one flaky provider never blocks results"],"tags":["third-party-api","crt-sh","subdomains","rate-limit"],"backgroundTag":"third-party-api-unexpected-response","analyzedSha":"af1a97759fc8bcc43c876c94f2ccb018ce215f90","analyzedAt":"2026-08-27T11:44:27.410Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}