{"record":{"id":"3c0563331054b6f6","repo":"jackwener/OpenCLI","slug":"substack-post-search-failed-http-resp-status","errorCode":null,"errorMessage":"Substack post search failed: HTTP ${resp.status}","messagePattern":"Substack post search failed: HTTP (.+?)","errorType":"exception","errorClass":"CommandExecutionError","httpStatus":null,"severity":"error","filePath":"clis/substack/search.js","lineNumber":26,"sourceCode":"}\nfunction trim(value) {\n    return typeof value === 'string' ? value.replace(/\\s+/g, ' ').trim() : '';\n}\nfunction publicationBaseUrl(publication) {\n    if (publication?.custom_domain)\n        return `https://${publication.custom_domain}`;\n    if (publication?.subdomain)\n        return `https://${publication.subdomain}.substack.com`;\n    return '';\n}\nasync function searchPosts(keyword, limit) {\n    const url = new URL('https://substack.com/api/v1/post/search');\n    url.searchParams.set('query', keyword);\n    url.searchParams.set('page', '0');\n    url.searchParams.set('includePlatformResults', 'true');\n    const resp = await fetch(url, { headers: headers() });\n    if (!resp.ok)\n        throw new CommandExecutionError(`Substack post search failed: HTTP ${resp.status}`);\n    const data = await resp.json();\n    const results = Array.isArray(data?.results) ? data.results : [];\n    return results.slice(0, limit).map((item, index) => ({\n        rank: index + 1,\n        title: trim(item?.title),\n        author: trim(item?.publishedBylines?.[0]?.name),\n        date: trim(item?.post_date).split('T')[0] || trim(item?.post_date),\n        description: trim(item?.description || item?.subtitle || item?.truncated_body_text).slice(0, 150),\n        url: trim(item?.canonical_url),\n    }));\n}\nasync function searchPublications(keyword, limit) {\n    const url = new URL('https://substack.com/api/v1/profile/search');\n    url.searchParams.set('query', keyword);\n    url.searchParams.set('page', '0');\n    const resp = await fetch(url, { headers: headers() });\n    if (!resp.ok)\n        throw new CommandExecutionError(`Substack publication search failed: HTTP ${resp.status}`);","sourceCodeStart":8,"sourceCodeEnd":44,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/substack/search.js#L8-L44","documentation":"searchPosts calls the Substack post search API (substack.com/api/v1/post/search). If the response is not ok, it throws a CommandExecutionError with the HTTP status, since the search could not complete.","triggerScenarios":"Any searchPosts call where substack.com returns a non-2xx status: rate limiting (429), auth/cookie rejection (401/403), server errors (5xx), or 404 when the endpoint changes.","commonSituations":"Heavy scraping triggering Substack rate limits, missing or expired browser session cookies expected by headers(), Substack API changes breaking the endpoint, datacenter IPs blocked by bot protection.","solutions":["Check the HTTP status in the message and retry with backoff for 429/5xx","Use an authenticated browser session so headers() carries valid cookies","Slow down request rate and add delays between searches","Verify the endpoint still exists if you get repeated 404s after a Substack update"],"exampleFix":"// before\nconst posts = await searchPosts('rust', 10); // throws on 429\n// after\nlet posts;\ntry { posts = await searchPosts('rust', 10); }\ncatch (e) { console.warn(e.message); await sleep(5000); posts = await searchPosts('rust', 10); }","handlingStrategy":"retry","validationCode":"// ensure a browser session exists so headers() carries valid cookies\nif (!session || !session.cookies) throw new Error('run within a browser session for substack search');","typeGuard":"null","tryCatchPattern":"try { posts = await searchPosts(q, 10); } catch (e) {\n  const m = /HTTP (\\d+)/.exec(e.message);\n  if (m && ['429','502','503'].includes(m[1])) { await sleep(5000); posts = await searchPosts(q, 10); } else throw e;\n}","preventionTips":["Add backoff on 429/5xx statuses","Use an authenticated browser session rather than bare fetches","Throttle search volume in loops","Watch for Substack API changes after 404s"],"tags":["http","search","substack"],"backgroundTag":"http-request-failed","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}