apify/crawlee · error · Error

${status} - Internal Server Error: ${body.slice(0, 100)}

Error message

${status} - Internal Server Error: ${body.slice(0, 100)}

What it means

For error-status responses with a non-JSON, non-HTML/XML content type, parseResponse throws `${status} - Internal Server Error: ${body.slice(0, 100)}`, including the first 100 characters of the body for diagnosis. This is the fallback so plain-text error payloads still surface their status and a snippet.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:745

        if (this.isErrorStatusCode(status)) {
            const body = await reencodedResponse.text(); // TODO - this always uses UTF-8 (see https://developer.mozilla.org/en-US/docs/Web/API/Request/text)

            // Errors are often sent as JSON, so attempt to parse them,
            // despite Accept header being set to text/html.
            if (type === APPLICATION_JSON_MIME_TYPE) {
                const errorResponse = JSON.parse(body);
                let { message } = errorResponse;
                if (!message) message = util.inspect(errorResponse, { depth: 1, maxArrayLength: 10 });
                throw new Error(`${status} - ${message}`);
            }

            if (this.additionalHttpErrorStatusCodes.has(status)) {
                throw new Error(`${status} - Error status code was set by user.`);
            }

            // It's not a JSON, so it's probably some text. Get the first 100 chars of it.
            throw new Error(`${status} - Internal Server Error: ${body.slice(0, 100)}`);
        } else if (HTML_AND_XML_MIME_TYPES.includes(type)) {
            if (!charset && !this.#forceResponseEncoding) {
                const rawBytes = Buffer.from(await response.arrayBuffer());
                const metaCharset = extractCharsetFromHtmlBytes(rawBytes);
                const charsetToUse = metaCharset ?? this.#suggestResponseEncoding ?? 'utf-8';
                const body = iconv.encodingExists(charsetToUse)
                    ? iconv.decode(rawBytes, charsetToUse)
                    : rawBytes.toString('utf8');
                return { response, contentType: { type, encoding: 'utf-8' as BufferEncoding }, body };
            }
            return { response, contentType, body: await reencodedResponse.text() };
        } else {
            const body = Buffer.from(await reencodedResponse.bytes());
            return {
                body,
                response,
                contentType,
            };

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Read the body snippet in the message — it usually names the real server-side problem to fix or report to the site owner.
2. Enable retries so transient 5xx responses are re-attempted later.
3. If such statuses are expected, route them via failedRequestHandler without treating the crawl as broken.
4. Consider whether a browser crawler is needed if the site serves challenges as plain text.

Example fix

// before
failedRequestHandler: ({ request }) => log.error(`Failed: ${request.url}`),

// after
failedRequestHandler: ({ request, error }) => {
  if (error && error.message.includes('Internal Server Error')) {
    log.warning(`Transient 5xx for ${request.url}: ${error.message}`);
  } else {
    log.error(`Failed: ${request.url}`);
  }
},

const res = await got(url, { throwHttpErrors: false });
if (res.statusCode >= 500 && !res.headers['content-type']?.includes('json')) {
  log.warning(`Plain-text 5xx: ${res.body.slice(0, 100)}`);
}

function isPlainTextServerError(err: unknown): boolean {
  return err instanceof Error && err.message.includes(' - Internal Server Error: ');
}

try {
  await crawler.run(requests);
} catch (err) {
  if (err instanceof Error && err.message.includes('Internal Server Error')) {
    log.warning(`Transient server error, snackbar: ${err.message.slice(0, 140)}`);
  } else {
    throw err;
  }
}

When it happens

Trigger: Server returns status >= 500 (or a configured blocked status) with Content-Type like text/plain, and the body is not valid JSON.

Common situations: Backend stack traces returned as text/plain on 500; gateways returning plain 'Bad Gateway' text; maintenance-mode pages; rate limiter responses as plain text.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• ${status} - ${message}
• ${status} - Error status code was set by user.
• Could not parse cookie header string: ${cookieHeaderString}
• Request with GET method cannot have a payload.
• Cannot fetch a request list from ${requestsFromUrl}: ${err}

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/7e3574b519d68451. Report an issue: GitHub.