{"record":{"id":"cf37b7efdd746ba2","repo":"apify/crawlee","slug":"status-error-status-code-was-set-by-user-cf37b7","errorCode":null,"errorMessage":"${status} - Error status code was set by user.","messagePattern":"(.+?) - Error status code was set by user\\.","errorType":"http","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":741,"sourceCode":"\n        if (status >= 400 && status <= 599) {\n            this.statistics.registerStatusCode(status);\n        }\n\n        if (this.isErrorStatusCode(status)) {\n            const body = await reencodedResponse.text(); // TODO - this always uses UTF-8 (see https://developer.mozilla.org/en-US/docs/Web/API/Request/text)\n\n            // Errors are often sent as JSON, so attempt to parse them,\n            // despite Accept header being set to text/html.\n            if (type === APPLICATION_JSON_MIME_TYPE) {\n                const errorResponse = JSON.parse(body);\n                let { message } = errorResponse;\n                if (!message) message = util.inspect(errorResponse, { depth: 1, maxArrayLength: 10 });\n                throw new Error(`${status} - ${message}`);\n            }\n\n            if (this.additionalHttpErrorStatusCodes.has(status)) {\n                throw new Error(`${status} - Error status code was set by user.`);\n            }\n\n            // It's not a JSON, so it's probably some text. Get the first 100 chars of it.\n            throw new Error(`${status} - Internal Server Error: ${body.slice(0, 100)}`);\n        } else if (HTML_AND_XML_MIME_TYPES.includes(type)) {\n            if (!charset && !this.#forceResponseEncoding) {\n                const rawBytes = Buffer.from(await response.arrayBuffer());\n                const metaCharset = extractCharsetFromHtmlBytes(rawBytes);\n                const charsetToUse = metaCharset ?? this.#suggestResponseEncoding ?? 'utf-8';\n                const body = iconv.encodingExists(charsetToUse)\n                    ? iconv.decode(rawBytes, charsetToUse)\n                    : rawBytes.toString('utf8');\n                return { response, contentType: { type, encoding: 'utf-8' as BufferEncoding }, body };\n            }\n            return { response, contentType, body: await reencodedResponse.text() };\n        } else {\n            const body = Buffer.from(await reencodedResponse.bytes());\n            return {","sourceCodeStart":723,"sourceCodeEnd":759,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L723-L759","documentation":"If the error status code was explicitly added by the user via `additionalHttpErrorStatusCodes`, parseResponse throws `${status} - Error status code was set by user.` because the response body is not JSON and no server message is available. This makes user-designated status codes fail the request even if they would otherwise be treated as success.","triggerScenarios":"Calling `new HttpCrawler({ additionalHttpErrorStatusCodes: [404, 410, ...] })` (or extendHttpCrawlerOptions equivalents) and receiving one of those statuses with a non-JSON body.","commonSituations":"Users marking soft-404 or bot-wall statuses as errors so they get retried; misremembering that additionalHttpErrorStatusCodes supplements (not replaces) the default error statuses; adding 2xx/3xx codes intentionally to force retries.","solutions":["Confirm the added status code is really one you want treated as an error; remove it from additionalHttpErrorStatusCodes if not.","Keep the code but handle it: check `error.message` in failedRequestHandler and skip logging as failure when expected.","Use `ignoreHttpErrors` for statuses you want passed through without retry instead of adding them as errors.","If the response is JSON, the first parse branch will produce a more descriptive message — no action needed."],"exampleFix":"// before\nnew HttpCrawler({ additionalHttpErrorStatusCodes: [403] });\n\n// after: treat 403 as blocked (session retry) instead of generic user error\nnew HttpCrawler({ retryOnBlocked: true, blockedStatusCodes: [403], useSessionPool: true });","handlingStrategy":"validation","validationCode":"const crawler = new HttpCrawler({ additionalHttpErrorStatusCodes: [404] });\n// audit config: only add statuses you truly want to fail requests\nconsole.log('Treating as errors:', [404]);","typeGuard":null,"tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('Error status code was set by user.')) {\n    const status = err.message.split(' - ')[0];\n    log.info(`Expected user-flagged status ${status}; ignoring`);\n    return;\n  }\n  throw err;\n}","preventionTips":["Document why each status is in additionalHttpErrorStatusCodes.","Use blockedStatusCodes for block/rate-limit statuses (enables session retry) instead of generic error codes.","Prefer ignoreHttpErrors over adding codes when the response is usable.","Review this option after upgrading crawlee, as defaults evolve."],"tags":["http","status-code","configuration","crawlee"],"backgroundTag":"http-error-status-code","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}