jackwener/OpenCLI · error · CommandExecutionError

lobsters domain returned malformed JSON: ${err?.message ?? e

Error message

lobsters domain returned malformed JSON: ${err?.message ?? err}

What it means

This CommandExecutionError is thrown when the HTTP response body from the lobste.rs domain endpoint cannot be parsed as JSON (resp.json() rejects). Since a non-ok status was already handled, this means the server returned a 2xx/3xx response whose body is not valid JSON — typically an HTML error/challenge page. The original parse error message is appended to help diagnose the malformed payload.

Source

Thrown at clis/lobsters/domain.js:73

        }
        catch (err) {
            throw new CommandExecutionError(
                `lobsters domain request failed: ${err?.message ?? err}`,
                'Check that lobste.rs is reachable from this network.',
            );
        }
        if (resp.status === 404) {
            throw new EmptyResultError('lobsters domain', `No Lobste.rs stories found for domain "${domain}".`);
        }
        if (!resp.ok) {
            throw new CommandExecutionError(`lobsters domain returned HTTP ${resp.status}`);
        }
        let body;
        try {
            body = await resp.json();
        }
        catch (err) {
            throw new CommandExecutionError(`lobsters domain returned malformed JSON: ${err?.message ?? err}`);
        }
        const list = Array.isArray(body) ? body : [];
        if (!list.length) {
            throw new EmptyResultError('lobsters domain', `No Lobste.rs stories found for domain "${domain}".`);
        }
        return list.slice(0, limit).map((item, i) => ({
            rank: i + 1,
            id: String(item.short_id ?? ''),
            title: String(item.title ?? ''),
            score: item.score != null ? Number(item.score) : null,
            author: String(item.submitter_user ?? ''),
            comments: item.comment_count != null ? Number(item.comment_count) : null,
            created_at: String(item.created_at ?? '').slice(0, 10),
            tags: Array.isArray(item.tags) ? item.tags.join(', ') : '',
            submission_url: String(item.url ?? ''),
            comments_url: String(item.comments_url ?? ''),
        }));
    },

View on GitHub (pinned to 49907e53dc)

Solutions

Fetch the endpoint manually (curl -i https://lobste.rs/domain/<domain>.json) and inspect whether the body is HTML from a challenge page or proxy.
Set a descriptive User-Agent header if you control the request, as Cloudflare frequently challenges requests with bot-like or missing User-Agents.
Retry from a different network (residential IP instead of datacenter/CI) to rule out Cloudflare/proxy interception.
Catch this error and surface the inner message plus a snippet of the raw body to confirm what was actually returned.

Example fix

// before
// resp.json() fails on a Cloudflare HTML page; you only see 'malformed JSON'

// after
const text = await resp.text();
let body;
try {
  body = JSON.parse(text);
} catch (err) {
  console.error('lobsters domain returned non-JSON body:', text.slice(0, 200));
  throw err;
}

Defensive patterns

Strategy: try-catch

Validate before calling

async function looksLikeJsonEndpoint(url) {
  try {
    const r = await fetch(url, { headers: { Accept: 'application/json' } });
    const ct = r.headers.get('content-type') || '';
    return ct.includes('application/json');
  } catch {
    return false;
  }
}

Type guard

null

Try / catch

try {
  const stories = await cli.lobsters.domain(domain);
} catch (err) {
  if (/malformed JSON/.test(String(err.message))) {
    console.error('lobste.rs returned a non-JSON body (likely Cloudflare challenge or proxy page). Try a different network or check https://lobste.rs directly.');
    return;
  }
  throw err;
}

Prevention

Check the response content-type is application/json before parsing in your own fetches.
Avoid calling from datacenter/CI IPs that commonly receive Cloudflare challenges; use a normal User-Agent.
Detect captive portals: if other HTTPS sites return HTML challenges, pause API calls.
Log a snippet of the raw body when JSON parsing fails to identify who injected the HTML.

When it happens

Trigger: lobste.rs (or an intermediary proxy/CDN) returns HTTP 200 with an HTML body: Cloudflare bot-challenge or 'checking your browser' page, a captive-portal login page, a maintenance page served with 200, or a truncated/corrupted response body.

Common situations: Running from datacenter IPs or CI runners that Cloudflare challenges, corporate proxies that rewrite responses, offline captive networks that intercept all HTTP with a login page, or a lobste.rs incident where the API serves HTML.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/017eec6696bcc8b8. Report an issue: GitHub.