What causes "init cache scheduler error" in code4craft/webmagic?

Create the target directory (Files.createDirectories) and verify write permission before constructing the scheduler.

code4craft/webmagic · critical · RuntimeException

init cache scheduler error

Error message

init cache scheduler error

What it means

FileCacheQueueScheduler's initWriter() opens the two persistence files (URL queue file and cursor file) with FileWriter. If either file cannot be opened (bad directory, permissions, path is a directory, disk issue), the IOException is wrapped in a RuntimeException with message "init cache scheduler error". This aborts scheduler initialization entirely.

Solutions

Create the target directory (Files.createDirectories) and verify write permission before constructing the scheduler.
Check that the scheduler path is a directory and that neither the .urls nor cursor file name collides with an existing directory.
Run the process as a user with write access to the folder, or move the scheduler folder to a writable location.
Catch the RuntimeException and fall back to the in-memory QueueScheduler if file persistence is optional.

Example fix

// before
scheduler = new FileCacheQueueScheduler("/app/data/queue");
// after
Files.createDirectories(Paths.get("/app/data/queue"));
scheduler = new FileCacheQueueScheduler("/app/data/queue");

Defensive patterns

Strategy: validation

Validate before calling

java.nio.file.Path dir = java.nio.file.Paths.get(schedulerPath);
if (!java.nio.file.Files.isDirectory(dir) || !java.nio.file.Files.isWritable(dir)) {
    throw new IllegalStateException("Scheduler dir missing or not writable: " + dir);
}

Try / catch

try {
    this.scheduler = new FileCacheQueueScheduler(path);
} catch (RuntimeException e) {
    log.warn("File scheduler init failed ({}), falling back to memory queue", e.getMessage());
    this.scheduler = new QueueScheduler();
}

Prevention

Always create the scheduler directory with Files.createDirectories before constructing
Verify write permissions of the crawler's data directory at startup
Use absolute paths for the scheduler folder to avoid working-directory surprises
Mount writable volumes in containers for persisted queue files

When it happens

Trigger: Calling PageRunner/startWithFileScheduler or constructing FileCacheQueueScheduler with a path where getFileName(fileUrlAllName) or getFileName(fileCursor) cannot be opened in append/write mode: parent directory missing, no write permission, path points to a directory, or filesystem errors.

Common situations: Passing a relative or non-existent directory to the file-scheduler path; running the crawler as a user without write access to the data folder; a directory named like the queue file already exists; read-only or full disk (e.g. in containers).

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of code4craft/webmagic@67816a19d6 (2026-09-08). Data as JSON: /api/errors/af174fdcac014592. Report an issue: GitHub.

Appendix: source

Thrown at webmagic-extension/src/main/java/us/codecraft/webmagic/scheduler/FileCacheQueueScheduler.java:82

        logger.info("init cache scheduler success");
    }

    private void initDuplicateRemover() {
        BloomFilterDuplicateRemover bloomFilterDuplicateRemover = new BloomFilterDuplicateRemover(this.filePath.hashCode());
        setDuplicateRemover(bloomFilterDuplicateRemover);
    }

    private void initFlushThread() {
        flushThreadPool = Executors.newScheduledThreadPool(1);
        flushThreadPool.scheduleAtFixedRate(this::flush, 10, 10, TimeUnit.SECONDS);
    }

    private void initWriter() {
        try {
            fileUrlWriter = new PrintWriter(new FileWriter(getFileName(fileUrlAllName), true));
            fileCursorWriter = new PrintWriter(new FileWriter(getFileName(fileCursor), false));
        } catch (IOException e) {
            throw new RuntimeException("init cache scheduler error", e);
        }
    }

    private void readFile() {
        try {
            queue = new LinkedBlockingQueue<Request>();
            readCursorFile();
            readUrlFile();
            // initDuplicateRemover();
        } catch (FileNotFoundException e) {
            //init
            logger.info("init cache file " + getFileName(fileUrlAllName));
        } catch (IOException e) {
            logger.error("init file error", e);
        }
    }

    private void readUrlFile() throws IOException {

View on GitHub (pinned to 67816a19d6)