{"record":{"id":"a5501eb2eee38d5a","repo":"apify/crawlee","slug":"status-message","errorCode":null,"errorMessage":"${status} - ${message}","messagePattern":"\\$\\{status\\} - \\$\\{message\\}","errorType":"http","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":737,"sourceCode":"        const { status } = response;\n        const { type, charset } = parseContentTypeFromResponse(response);\n        const { response: reencodedResponse, encoding } = this.encodeResponse(request, response, charset);\n        const contentType = { type, encoding };\n\n        if (status >= 400 && status <= 599) {\n            this.statistics.registerStatusCode(status);\n        }\n\n        if (this.isErrorStatusCode(status)) {\n            const body = await reencodedResponse.text(); // TODO - this always uses UTF-8 (see https://developer.mozilla.org/en-US/docs/Web/API/Request/text)\n\n            // Errors are often sent as JSON, so attempt to parse them,\n            // despite Accept header being set to text/html.\n            if (type === APPLICATION_JSON_MIME_TYPE) {\n                const errorResponse = JSON.parse(body);\n                let { message } = errorResponse;\n                if (!message) message = util.inspect(errorResponse, { depth: 1, maxArrayLength: 10 });\n                throw new Error(`${status} - ${message}`);\n            }\n\n            if (this.additionalHttpErrorStatusCodes.has(status)) {\n                throw new Error(`${status} - Error status code was set by user.`);\n            }\n\n            // It's not a JSON, so it's probably some text. Get the first 100 chars of it.\n            throw new Error(`${status} - Internal Server Error: ${body.slice(0, 100)}`);\n        } else if (HTML_AND_XML_MIME_TYPES.includes(type)) {\n            if (!charset && !this.#forceResponseEncoding) {\n                const rawBytes = Buffer.from(await response.arrayBuffer());\n                const metaCharset = extractCharsetFromHtmlBytes(rawBytes);\n                const charsetToUse = metaCharset ?? this.#suggestResponseEncoding ?? 'utf-8';\n                const body = iconv.encodingExists(charsetToUse)\n                    ? iconv.decode(rawBytes, charsetToUse)\n                    : rawBytes.toString('utf8');\n                return { response, contentType: { type, encoding: 'utf-8' as BufferEncoding }, body };\n            }","sourceCodeStart":719,"sourceCodeEnd":755,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L719-L755","documentation":"When a response has an error status code (>= 500 or in blockedStatusCodes / additionalHttpErrorStatusCodes handling path) and Content-Type application/json, parseResponse JSON-parses the body and throws `${status} - ${message}` using the JSON's `message` field (or util.inspect of the whole payload if absent). This surfaces the server's own error description to the retry/error machinery.","triggerScenarios":"The target responds with a non-2xx status (e.g. 500, 502, 503) and JSON body like {\"message\":\"upstream timeout\"}; the crawler's statusCode validation converts it into this thrown Error.","commonSituations":"API endpoints returning JSON errors for rate limiting or server faults; gateways (Cloudflare/nginx) emitting JSON error payloads; temporary upstream outages while crawling APIs.","solutions":["Inspect the embedded message to see what the server actually complained about and fix that root cause (payload, auth, rate).","Configure the crawler's retry settings so transient 5xx responses are retried with backoff.","If the status is actually acceptable for your use case, add it via `ignoreHttpErrors` / adjust `additionalHttpErrorStatusCodes` / `blockedStatusCodes`.","Implement `postNavigationHooks` or a custom requestFunction to handle known JSON error contracts gracefully."],"exampleFix":"// before\nnew HttpCrawler({ additionalHttpErrorStatusCodes: [429] });\n\n// after: accept 429 responses instead of throwing\nnew HttpCrawler({ ignoreHttpErrors: false, additionalHttpErrorStatusCodes: [], ignoreSslErrors: false,\n  // handle 429 by retrying later\n  failedRequestHandler: async ({ request, error }) => log.error(`${request.url}: ${error.message}`),\n});","handlingStrategy":"retry","validationCode":"const res = await got(url, { throwHttpErrors: false, responseType: 'json' });\nif (res.statusCode >= 500 && res.body?.message) {\n  log.warning(`Server reports: ${res.body.message}`);\n}","typeGuard":"function isStatusJsonError(err: unknown, status: number): boolean {\n  return err instanceof Error && err.message.startsWith(`${status} - `);\n}","tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  const m = /^\\d{3} - /.exec(err instanceof Error ? err.message : '');\n  if (m) {\n    log.warning(`HTTP error response, will rely on crawler retry: ${err.message}`);\n  } else {\n    throw err;\n  }\n}","preventionTips":["Keep maxRequestRetries > 0 so 5xx JSON errors are retried automatically.","Read the embedded server message first — it names the real cause.","Use ignoreHttpErrors for statuses that are actually acceptable.","Monitor error-rate per domain and back off when 5xx spikes occur."],"tags":["http","status-code","json","server-error"],"backgroundTag":"http-5xx-response","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}