lissy93/web-check · warning · Error

crt.sh returned an unexpected response

Error message

crt.sh returned an unexpected response

What it means

crtSh fetches crt.sh's JSON issuance feed and expects an array; a non-array body (HTML error page, rate-limit text, empty body on crt.sh's frequent 503s) triggers this throw. crt.sh is a free, unauthenticated service with known instability under load.

Source

Thrown at api/subdomains.js:30

const certSpotter = async (domain) => {
  const token = process.env.CERTSPOTTER_TOKEN;
  const res = await httpGet('https://api.certspotter.com/v1/issuances', {
    params: { domain, include_subdomains: 'true', expand: 'dns_names' },
    headers: { Accept: 'application/json', ...(token && { Authorization: `Bearer ${token}` }) },
    timeout: SOURCE_TIMEOUT,
  });
  if (!Array.isArray(res.data)) throw new Error('certSpotter returned an unexpected response');
  return res.data.flatMap((row) => (Array.isArray(row?.dns_names) ? row.dns_names : []));
};

const crtSh = async (domain) => {
  const res = await httpGet('https://crt.sh/', {
    params: { q: `%.${domain}`, output: 'json' },
    headers: { Accept: 'application/json' },
    timeout: SOURCE_TIMEOUT,
  });
  if (!Array.isArray(res.data)) throw new Error('crt.sh returned an unexpected response');
  return res.data.flatMap((row) => String(row?.name_value ?? '').split('\n'));
};

const hackerTarget = async (domain) => {
  const res = await httpGet('https://api.hackertarget.com/hostsearch/', {
    params: { q: domain },
    timeout: SOURCE_TIMEOUT,
  });
  const body = typeof res.data === 'string' ? res.data : '';
  if (!body || /error|api count|quota/i.test(body)) throw new Error('hackerTarget unavailable');
  return body.split('\n').map((line) => line.split(',')[0]);
};

const SOURCES = [
  { name: 'certSpotter', lookup: certSpotter },
  { name: 'crt.sh', lookup: crtSh },
  { name: 'hackerTarget', lookup: hackerTarget },
];

View on GitHub (pinned to af1a97759f)

Solutions

1. Treat crt.sh as best-effort: catch per-source and continue with other SOURCES (aggregation already exists)
2. Throttle/retry crt.sh with exponential backoff (it is heavily rate-limited)
3. If persistent, temporarily remove crt.sh from SOURCES until it recovers

Example fix

// before
const found = await crtSh(domain); // single-source dependency

// after
const safe = async (fn, name) => { try { return await fn(domain); } catch (e) { console.warn(`${name} unavailable: ${e.message}`); return []; } };
const found = [...await safe(certSpotter, 'certSpotter'), ...await safe(crtSh, 'crt.sh'), ...await safe(hackerTarget, 'hackerTarget')];

null

null

const withRetry = async (fn, tries = 3) => {
  for (let i = 0; i < tries; i++) {
    try { return await fn(domain); }
    catch (e) { if (i === tries - 1) { console.warn(`crt.sh failed: ${e.message}`); return []; } await new Promise(r => setTimeout(r, 2 ** i * 1000)); }
  }
};

When it happens

Trigger: crt.sh returns its common 'Server Error'/503 HTML page, rate-limits the caller, or returns a JSON error object instead of the issuance array.

Common situations: Bulk subdomain scans hitting crt.sh too fast, crt.sh being intermittently down (well-known), or timeout truncation producing a partial/empty body.

Related errors

• certSpotter returned an unexpected response
• hackerTarget unavailable

AI-assisted analysis of lissy93/web-check@af1a97759f (2026-08-27). Data as JSON: /api/errors/5873b07e3d2686bf. Report an issue: GitHub.