apify/crawlee · warning · ContextPipelineInterruptedError

Skipping request ${request.id} (starting url: ${request.url}

Error message

Skipping request ${request.id} (starting url: ${request.url} -> loaded url: ${request.loadedUrl}) because it does not match the enqueue strategy (${request['enqueueStrategy']}).

What it means

After a redirect, the crawler checks whether the final loaded URL still satisfies the request's enqueue strategy (e.g. SAME_DOMAIN, SAME_HOSTNAME). If the redirect crossed outside the allowed scope, the request is marked SKIPPED, recorded as skipped with reason 'redirect', and the pipeline is interrupted with this error.

Source

Thrown at packages/basic-crawler/src/internals/basic-crawler.ts:1594

                ) as unknown as ContextPipeline<CrawlingContext, Context>;
        } else {
            contextPipeline = subclassPipeline;
        }

        contextPipeline = contextPipeline.compose({
            action: async (context) => {
                const { request } = context;
                if (request && !this.requestMatchesEnqueueStrategy(request)) {
                    // eslint-disable-next-line dot-notation
                    const message = `Skipping request ${request.id} (starting url: ${request.url} -> loaded url: ${request.loadedUrl}) because it does not match the enqueue strategy (${request['enqueueStrategy']}).`;
                    this.log.debug(message);

                    request.noRetry = true;
                    request.state = RequestState.SKIPPED;

                    await this.#handleSkippedRequest({ request, reason: 'redirect' });

                    throw new ContextPipelineInterruptedError(message);
                }
                return context;
            },
        });

        return contextPipeline as ContextPipeline<CrawlingContext, ExtendedContext>;
    }

    /**
     * Checks if the given error is a proxy error by comparing its message to a list of known proxy error messages.
     * Used for retrying requests that failed due to proxy errors.
     *
     * @param error The error to check.
     */
    protected isProxyError(error: Error): boolean {
        return ROTATE_PROXY_ERRORS.some((x: string) => (this.getMessageFromError(error) as any)?.includes(x));
    }

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Loosen the enqueue strategy (e.g. use EnqueueStrategy.SameDomain instead of SameHostname) if the redirects are legitimate.
2. Pre-resolve redirects with a HEAD/GET request and enqueue the final URL directly.
3. Handle the skipped 'redirect' requests in the skipped-request hook and re-enqueue the loadedUrl with an appropriate strategy.
4. Add the redirect target's domain to the allowed domains in strategy options if it should be followed.

Example fix

// before
const crawler = new CheerioCrawler({ enqueueStrategy: { strategy: 'same-hostname' } });
// after
const crawler = new CheerioCrawler({ enqueueStrategy: { strategy: 'same-domain' } });

// Verify the final URL of a request stays inside the strategy before enqueueing
const finalUrl = await resolveRedirects(request.url); // HEAD-follow helper
if (!isSameDomain(finalUrl, new URL(request.url).hostname)) {
  request.enqueueStrategy = { strategy: 'same-domain' }; // loosen or skip enqueueing
}

function matchesStrategy(loadedUrl, originUrl, strategy) {
  const a = new URL(loadedUrl), b = new URL(originUrl);
  if (strategy === 'same-hostname') return a.hostname === b.hostname;
  if (strategy === 'same-domain') return a.hostname.replace(/^www\./, '') === b.hostname.replace(/^www\./, '');
  return true;
}

try {
  await crawler.run(requests);
} catch (err) {
  if (err.name === 'ContextPipelineInterruptedError' && err.message.includes('enqueue strategy')) {
    // treat request as skipped; optionally re-enqueue request.loadedUrl with a looser strategy
  } else throw err;
}

When it happens

Trigger: A request enqueued with a restrictive enqueueStrategy (e.g. sameHostname) whose server responds with a redirect to a URL that violates that strategy — the loaded URL no longer matches, so the request is skipped.

Common situations: Sites redirecting http->a different subdomain, to a CDN domain, or to a country-specific domain; enqueuing with SAME_HOSTNAME and hitting a www/non-www or short-link redirect.

Related errors

• Skipping request ${request.url} as disallowed by robots.txt
• Too many redirects (${maxRedirects}) while requesting ${curr
• Unknown enqueue strategy '${strategy satisfies never}'.

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/8a790fe18acdb409. Report an issue: GitHub.