apache/cassandra · error · IllegalStateException

HintsService is shut down and cannot be restarted

Error message

HintsService is shut down and cannot be restarted

What it means

Fired in HintsService.startDispatch when isShutDown is true: dispatch of hints is requested after the service was shut down (typically during node shutdown/drain), so restarting it is illegal. An IllegalStateException is thrown to protect the drained state; only full service re-initialization (node restart) can restore dispatch.

Source

Thrown at src/java/org/apache/cassandra/hints/HintsService.java:254

    public void flushAndFsyncBlockingly(Iterable<UUID> hostIds)
    {
        Iterable<HintsStore> stores = filter(transform(hostIds, catalog::getNullable), Objects::nonNull);
        writeExecutor.flushBufferPool(bufferPool, stores);
        writeExecutor.fsyncWritersBlockingly(stores);
    }

    @VisibleForTesting
    public void flushAndFsyncBlockingly()
    {
        List<HintsStore> stores = catalog.stores().collect(Collectors.toList());
        writeExecutor.flushBufferPool(bufferPool, stores);
        writeExecutor.fsyncWritersBlockingly(stores);
    }

    public synchronized void startDispatch()
    {
        if (isShutDown)
            throw new IllegalStateException("HintsService is shut down and cannot be restarted");

        isDispatchPaused.set(false);

        HintsServiceDiagnostics.dispatchingStarted(this);

        HintsDispatchTrigger trigger = new HintsDispatchTrigger(catalog, writeExecutor, dispatchExecutor, isDispatchPaused);
        // triggering hint dispatch is now very cheap, so we can do it more often - every 10 seconds vs. every 10 minutes,
        // previously; this reduces mean time to delivery, and positively affects batchlog delivery latencies, too
        long hintDispatchIntervalMs = HINT_DISPATCH_INTERVAL_MS.getLong();
        triggerDispatchFuture = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(trigger, hintDispatchIntervalMs, hintDispatchIntervalMs, TimeUnit.MILLISECONDS);
    }

    public void pauseDispatch()
    {
        logger.info("Paused hints dispatch");
        isDispatchPaused.set(true);

        HintsServiceDiagnostics.dispatchingPaused(this);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Do not call startDispatch() once shutdown has begun; guard with a service-lifecycle check
2. Catch IllegalStateException and treat as a no-op (dispatch is intentionally stopped)
3. Reorder lifecycle code so pause/resume dispatch happens before shutdownBlocking()
4. If the node needs hints again, restart the node rather than restarting the service

Example fix

// before
HintsService.instance.startDispatch();
// after
if (!HintsService.instance.isShutDown()) {
    HintsService.instance.startDispatch();
}

if (HintsService.instance.isShutDown()) {
    logger.warn("startDispatch ignored: service shut down");
    return;
}

boolean canStartDispatch() { return !HintsService.instance.isShutDown(); }

try {
    HintsService.instance.startDispatch();
} catch (IllegalStateException e) {
    logger.warn("Dispatch cannot start after shutdown", e);
}

When it happens

Trigger: Calling HintsService.startDispatch() after shutdownBlocking(); e.g. calling it in response to a gossiper/event after node shutdown has begun, or JMX-triggered startDispatch on a stopping node.

Common situations: Node shutdown racing with a dispatch pause/resume cycle; scripts or JMX clients issuing startDispatch while the node is stopping; test harness lifecycle ordering mistakes.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• HintsService is shut down and can't accept new hints
• HintsService has already been shut down
• Scheduler has shut down.
• Hint dispatch was interrupted
• Some hints were not written before shutdown. This is not su

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/958299047e006b74. Report an issue: GitHub.