{"record":{"id":"958299047e006b74","repo":"apache/cassandra","slug":"hintsservice-is-shut-down-and-cannot-be-restarted","errorCode":null,"errorMessage":"HintsService is shut down and cannot be restarted","messagePattern":"HintsService is shut down and cannot be restarted","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/hints/HintsService.java","lineNumber":254,"sourceCode":"    public void flushAndFsyncBlockingly(Iterable<UUID> hostIds)\n    {\n        Iterable<HintsStore> stores = filter(transform(hostIds, catalog::getNullable), Objects::nonNull);\n        writeExecutor.flushBufferPool(bufferPool, stores);\n        writeExecutor.fsyncWritersBlockingly(stores);\n    }\n\n    @VisibleForTesting\n    public void flushAndFsyncBlockingly()\n    {\n        List<HintsStore> stores = catalog.stores().collect(Collectors.toList());\n        writeExecutor.flushBufferPool(bufferPool, stores);\n        writeExecutor.fsyncWritersBlockingly(stores);\n    }\n\n    public synchronized void startDispatch()\n    {\n        if (isShutDown)\n            throw new IllegalStateException(\"HintsService is shut down and cannot be restarted\");\n\n        isDispatchPaused.set(false);\n\n        HintsServiceDiagnostics.dispatchingStarted(this);\n\n        HintsDispatchTrigger trigger = new HintsDispatchTrigger(catalog, writeExecutor, dispatchExecutor, isDispatchPaused);\n        // triggering hint dispatch is now very cheap, so we can do it more often - every 10 seconds vs. every 10 minutes,\n        // previously; this reduces mean time to delivery, and positively affects batchlog delivery latencies, too\n        long hintDispatchIntervalMs = HINT_DISPATCH_INTERVAL_MS.getLong();\n        triggerDispatchFuture = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(trigger, hintDispatchIntervalMs, hintDispatchIntervalMs, TimeUnit.MILLISECONDS);\n    }\n\n    public void pauseDispatch()\n    {\n        logger.info(\"Paused hints dispatch\");\n        isDispatchPaused.set(true);\n\n        HintsServiceDiagnostics.dispatchingPaused(this);","sourceCodeStart":236,"sourceCodeEnd":272,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/hints/HintsService.java#L236-L272","documentation":"Fired in HintsService.startDispatch when isShutDown is true: dispatch of hints is requested after the service was shut down (typically during node shutdown/drain), so restarting it is illegal. An IllegalStateException is thrown to protect the drained state; only full service re-initialization (node restart) can restore dispatch.","triggerScenarios":"Calling HintsService.startDispatch() after shutdownBlocking(); e.g. calling it in response to a gossiper/event after node shutdown has begun, or JMX-triggered startDispatch on a stopping node.","commonSituations":"Node shutdown racing with a dispatch pause/resume cycle; scripts or JMX clients issuing startDispatch while the node is stopping; test harness lifecycle ordering mistakes.","solutions":["Do not call startDispatch() once shutdown has begun; guard with a service-lifecycle check","Catch IllegalStateException and treat as a no-op (dispatch is intentionally stopped)","Reorder lifecycle code so pause/resume dispatch happens before shutdownBlocking()","If the node needs hints again, restart the node rather than restarting the service"],"exampleFix":"// before\nHintsService.instance.startDispatch();\n// after\nif (!HintsService.instance.isShutDown()) {\n    HintsService.instance.startDispatch();\n}","handlingStrategy":"try-catch","validationCode":"if (HintsService.instance.isShutDown()) {\n    logger.warn(\"startDispatch ignored: service shut down\");\n    return;\n}","typeGuard":"boolean canStartDispatch() { return !HintsService.instance.isShutDown(); }","tryCatchPattern":"try {\n    HintsService.instance.startDispatch();\n} catch (IllegalStateException e) {\n    logger.warn(\"Dispatch cannot start after shutdown\", e);\n}","preventionTips":["Only toggle dispatch pause/resume while the node is live","Guard JMX/scripts that call startDispatch with lifecycle checks","Order shutdown: pause dispatch -> shutdownBlocking","Restart the whole node if hints dispatch is needed after shutdown"],"tags":["lifecycle","shutdown","dispatch","hints"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}