{"record":{"id":"7e3574b519d68451","repo":"apify/crawlee","slug":"status-internal-server-error-body-slice-0","errorCode":null,"errorMessage":"${status} - Internal Server Error: ${body.slice(0, 100)}","messagePattern":"(.+?) - Internal Server Error: (.+?)","errorType":"http","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":745,"sourceCode":"\n        if (this.isErrorStatusCode(status)) {\n            const body = await reencodedResponse.text(); // TODO - this always uses UTF-8 (see https://developer.mozilla.org/en-US/docs/Web/API/Request/text)\n\n            // Errors are often sent as JSON, so attempt to parse them,\n            // despite Accept header being set to text/html.\n            if (type === APPLICATION_JSON_MIME_TYPE) {\n                const errorResponse = JSON.parse(body);\n                let { message } = errorResponse;\n                if (!message) message = util.inspect(errorResponse, { depth: 1, maxArrayLength: 10 });\n                throw new Error(`${status} - ${message}`);\n            }\n\n            if (this.additionalHttpErrorStatusCodes.has(status)) {\n                throw new Error(`${status} - Error status code was set by user.`);\n            }\n\n            // It's not a JSON, so it's probably some text. Get the first 100 chars of it.\n            throw new Error(`${status} - Internal Server Error: ${body.slice(0, 100)}`);\n        } else if (HTML_AND_XML_MIME_TYPES.includes(type)) {\n            if (!charset && !this.#forceResponseEncoding) {\n                const rawBytes = Buffer.from(await response.arrayBuffer());\n                const metaCharset = extractCharsetFromHtmlBytes(rawBytes);\n                const charsetToUse = metaCharset ?? this.#suggestResponseEncoding ?? 'utf-8';\n                const body = iconv.encodingExists(charsetToUse)\n                    ? iconv.decode(rawBytes, charsetToUse)\n                    : rawBytes.toString('utf8');\n                return { response, contentType: { type, encoding: 'utf-8' as BufferEncoding }, body };\n            }\n            return { response, contentType, body: await reencodedResponse.text() };\n        } else {\n            const body = Buffer.from(await reencodedResponse.bytes());\n            return {\n                body,\n                response,\n                contentType,\n            };","sourceCodeStart":727,"sourceCodeEnd":763,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L727-L763","documentation":"For error-status responses with a non-JSON, non-HTML/XML content type, parseResponse throws `${status} - Internal Server Error: ${body.slice(0, 100)}`, including the first 100 characters of the body for diagnosis. This is the fallback so plain-text error payloads still surface their status and a snippet.","triggerScenarios":"Server returns status >= 500 (or a configured blocked status) with Content-Type like text/plain, and the body is not valid JSON.","commonSituations":"Backend stack traces returned as text/plain on 500; gateways returning plain 'Bad Gateway' text; maintenance-mode pages; rate limiter responses as plain text.","solutions":["Read the body snippet in the message — it usually names the real server-side problem to fix or report to the site owner.","Enable retries so transient 5xx responses are re-attempted later.","If such statuses are expected, route them via failedRequestHandler without treating the crawl as broken.","Consider whether a browser crawler is needed if the site serves challenges as plain text."],"exampleFix":"// before\nfailedRequestHandler: ({ request }) => log.error(`Failed: ${request.url}`),\n\n// after\nfailedRequestHandler: ({ request, error }) => {\n  if (error && error.message.includes('Internal Server Error')) {\n    log.warning(`Transient 5xx for ${request.url}: ${error.message}`);\n  } else {\n    log.error(`Failed: ${request.url}`);\n  }\n},","handlingStrategy":"retry","validationCode":"const res = await got(url, { throwHttpErrors: false });\nif (res.statusCode >= 500 && !res.headers['content-type']?.includes('json')) {\n  log.warning(`Plain-text 5xx: ${res.body.slice(0, 100)}`);\n}","typeGuard":"function isPlainTextServerError(err: unknown): boolean {\n  return err instanceof Error && err.message.includes(' - Internal Server Error: ');\n}","tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('Internal Server Error')) {\n    log.warning(`Transient server error, snackbar: ${err.message.slice(0, 140)}`);\n  } else {\n    throw err;\n  }\n}","preventionTips":["Treat these as transient: rely on built-in retries with backoff.","The 100-char body snippet in the message is your best diagnostic — capture it in logs.","Check target site status page / health endpoint during crawl outages.","Alert on sustained 5xx rates rather than failing whole crawls."],"tags":["http","status-code","server-error","plain-text"],"backgroundTag":"http-5xx-response","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}