code4craft/webmagic · error · IllegalStateException

Spider is already running!

Error message

Spider is already running!

What it means

Spider.checkRunningStat() uses an AtomicInteger stat guard to ensure a Spider instance only runs once. If another thread (or an overlapping call) has already moved the spider into STAT_RUNNING, it throws IllegalStateException instead of starting a duplicate crawl.

Solutions

  1. Ensure only one invocation: call run() OR runAsync() once per Spider instance
  2. Create a new Spider (e.g. spider.clone() or WebMagic spider factory) for each crawl instead of reusing the running one
  3. Call stop() and wait for the previous run to fully finish before restarting
  4. Check spider.getStatus() before starting a run

Example fix

// before
spider.run(); // second invocation on same spider while running -> IllegalStateException
// after
if (spider.getStatus() == Status.STOPPED) {
    spider.run();
} else {
    spider.clone().run();
}
Defensive patterns

Strategy: try-catch

Validate before calling

if (spider.getStatus() == Status.RUNNING) { throw ... or create a new spider }

Try / catch

try { spider.run(); } catch (IllegalStateException e) { /* spider busy: clone or reuse existing run */ }

Prevention

When it happens

Trigger: Calling spider.run() (or runAsync/thread pool execution) on the same Spider instance that is already executing, or on a spider whose stat was left at STAT_RUNNING after an unclean stop.

Common situations: Re-invoking run() after runAsync(); sharing one Spider object across multiple threads; wrapping run() in a loop that restarts without calling stop(); a previous run crashed before stat returned to STAT_STOPPED.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of code4craft/webmagic@67816a19d6 (2026-09-08). Data as JSON: /api/errors/542e865e29c8e501. Report an issue: GitHub.

Appendix: source

Thrown at webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java:392

            for (SpiderListener spiderListener : spiderListeners) {
                spiderListener.onError(request, e);
            }
        }
    }

    protected void onSuccess(Request request) {
        if (CollectionUtils.isNotEmpty(spiderListeners)) {
            for (SpiderListener spiderListener : spiderListeners) {
                spiderListener.onSuccess(request);
            }
        }
    }

    private void checkRunningStat() {
        while (true) {
            int statNow = stat.get();
            if (statNow == STAT_RUNNING) {
                throw new IllegalStateException("Spider is already running!");
            }
            if (stat.compareAndSet(statNow, STAT_RUNNING)) {
                break;
            }
        }
    }

    public void close() {
        destroyEach(downloader);
        destroyEach(pageProcessor);
        destroyEach(scheduler);
        for (Pipeline pipeline : pipelines) {
            destroyEach(pipeline);
        }
        threadPool.shutdown();
    }

    private void destroyEach(Object object) {

View on GitHub (pinned to 67816a19d6)