{"record":{"id":"af174fdcac014592","repo":"code4craft/webmagic","slug":"init-cache-scheduler-error","errorCode":null,"errorMessage":"init cache scheduler error","messagePattern":"init cache scheduler error","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"webmagic-extension/src/main/java/us/codecraft/webmagic/scheduler/FileCacheQueueScheduler.java","lineNumber":82,"sourceCode":"        logger.info(\"init cache scheduler success\");\n    }\n\n    private void initDuplicateRemover() {\n        BloomFilterDuplicateRemover bloomFilterDuplicateRemover = new BloomFilterDuplicateRemover(this.filePath.hashCode());\n        setDuplicateRemover(bloomFilterDuplicateRemover);\n    }\n\n    private void initFlushThread() {\n        flushThreadPool = Executors.newScheduledThreadPool(1);\n        flushThreadPool.scheduleAtFixedRate(this::flush, 10, 10, TimeUnit.SECONDS);\n    }\n\n    private void initWriter() {\n        try {\n            fileUrlWriter = new PrintWriter(new FileWriter(getFileName(fileUrlAllName), true));\n            fileCursorWriter = new PrintWriter(new FileWriter(getFileName(fileCursor), false));\n        } catch (IOException e) {\n            throw new RuntimeException(\"init cache scheduler error\", e);\n        }\n    }\n\n    private void readFile() {\n        try {\n            queue = new LinkedBlockingQueue<Request>();\n            readCursorFile();\n            readUrlFile();\n            // initDuplicateRemover();\n        } catch (FileNotFoundException e) {\n            //init\n            logger.info(\"init cache file \" + getFileName(fileUrlAllName));\n        } catch (IOException e) {\n            logger.error(\"init file error\", e);\n        }\n    }\n\n    private void readUrlFile() throws IOException {","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/code4craft/webmagic/blob/67816a19d68a4fec4657bf1336227e046e251df2/webmagic-extension/src/main/java/us/codecraft/webmagic/scheduler/FileCacheQueueScheduler.java#L64-L100","documentation":"FileCacheQueueScheduler's initWriter() opens the two persistence files (URL queue file and cursor file) with FileWriter. If either file cannot be opened (bad directory, permissions, path is a directory, disk issue), the IOException is wrapped in a RuntimeException with message \"init cache scheduler error\". This aborts scheduler initialization entirely.","triggerScenarios":"Calling PageRunner/startWithFileScheduler or constructing FileCacheQueueScheduler with a path where getFileName(fileUrlAllName) or getFileName(fileCursor) cannot be opened in append/write mode: parent directory missing, no write permission, path points to a directory, or filesystem errors.","commonSituations":"Passing a relative or non-existent directory to the file-scheduler path; running the crawler as a user without write access to the data folder; a directory named like the queue file already exists; read-only or full disk (e.g. in containers).","solutions":["Create the target directory (Files.createDirectories) and verify write permission before constructing the scheduler.","Check that the scheduler path is a directory and that neither the .urls nor cursor file name collides with an existing directory.","Run the process as a user with write access to the folder, or move the scheduler folder to a writable location.","Catch the RuntimeException and fall back to the in-memory QueueScheduler if file persistence is optional."],"exampleFix":"// before\nscheduler = new FileCacheQueueScheduler(\"/app/data/queue\");\n// after\nFiles.createDirectories(Paths.get(\"/app/data/queue\"));\nscheduler = new FileCacheQueueScheduler(\"/app/data/queue\");","handlingStrategy":"validation","validationCode":"java.nio.file.Path dir = java.nio.file.Paths.get(schedulerPath);\nif (!java.nio.file.Files.isDirectory(dir) || !java.nio.file.Files.isWritable(dir)) {\n    throw new IllegalStateException(\"Scheduler dir missing or not writable: \" + dir);\n}","typeGuard":null,"tryCatchPattern":"try {\n    this.scheduler = new FileCacheQueueScheduler(path);\n} catch (RuntimeException e) {\n    log.warn(\"File scheduler init failed ({}), falling back to memory queue\", e.getMessage());\n    this.scheduler = new QueueScheduler();\n}","preventionTips":["Always create the scheduler directory with Files.createDirectories before constructing","Verify write permissions of the crawler's data directory at startup","Use absolute paths for the scheduler folder to avoid working-directory surprises","Mount writable volumes in containers for persisted queue files"],"tags":["java","io","scheduler","file-write"],"backgroundTag":"file-write-failed","analyzedSha":"67816a19d68a4fec4657bf1336227e046e251df2","analyzedAt":"2026-09-08T11:15:28.425Z","contentChangedAt":"2026-09-08T11:15:28.425Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}