What causes "Failed to transfer all hints to {}: {}; will retry in {} seconds" in apache/cassandra?

Confirm the target node is actually up and reachable (nodetool status, ping gossip port)

Failed to transfer all hints to

Error message

Failed to transfer all hints to {}: {}; will retry in {} seconds

What it means

HintsService's dispatch task transfers all accumulated hints to a target host (typically when a node is declared alive again after HINTED_HANDOFF or via 'nodetool truncat... / hint delivery'). If transfer(hostId) fails to deliver everything, the run loop logs this warning, sleeps 10 seconds, and retries until successful or cancelled.

Solutions

Confirm the target node is actually up and reachable (nodetool status, ping gossip port)
Check whether the hostId maps to a live endpoint; if the node was replaced/decommissioned, clear stale hints (nodetool truncatehints or delete hints files)
Inspect logs on both ends for messaging failures during the transfer window
If the target is gone permanently, remove pending hints for that hostId

Example fix

// before: retrying forever for a decommissioned host
// after: drop hints for the dead hostId
nodetool truncatehints --by-host <dead-host-uuid>  # or remove files in hints dir

Defensive patterns

Strategy: retry

Validate before calling

// before transfer, check the target is resolvable and alive
InetAddressAndPort ep = StorageService.instance.getEndpointForHostId(hostId);
boolean transferable = ep != null && FailureDetector.instance.isAlive(ep);

Try / catch

if (!transfer(hostId)) {
  logger.warn("Hints transfer to {} incomplete; retrying", hostId);
  TimeUnit.SECONDS.sleep(retryDelaySeconds);
}

Prevention

Verify target node health before expecting handoff to complete
Clean up hints for decommissioned/replaced hosts (truncatehints)
Watch for repeated warnings — indicates persistent connectivity issues
Ensure time synchronization and stable messaging between DCs

When it happens

Trigger: MaxHintsTransferAttempts-style run loop where transfer() returns false because the target endpoint is unavailable, the internode messaging fails, or the target hostId has no resolvable endpoint (getEndpointForHostId returns null).

Common situations: Target node flapping up/down during handoff; node decommissioned but its hostId still referenced; network instability between DCs during hinted handoff.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/db7c84bcf5a94898. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/hints/HintsDispatchExecutor.java:176

         */
        private final Supplier<UUID> hostIdSupplier;

        private TransferHintsTask(HintsCatalog catalog, Supplier<UUID> hostIdSupplier)
        {
            this.catalog = catalog;
            this.hostIdSupplier = hostIdSupplier;
        }

        @Override
        public void run()
        {
            UUID hostId = hostIdSupplier.get();
            InetAddressAndPort address = StorageService.instance.getEndpointForHostId(hostId);
            logger.info("Transferring all hints to {}: {}", address, hostId);
            if (transfer(hostId))
                return;

            logger.warn("Failed to transfer all hints to {}: {}; will retry in {} seconds", address, hostId, 10);

            try
            {
                TimeUnit.SECONDS.sleep(10);
            }
            catch (InterruptedException e)
            {
                throw new UncheckedInterruptedException(e);
            }

            hostId = hostIdSupplier.get();
            logger.info("Transferring all hints to {}: {}", address, hostId);
            if (!transfer(hostId))
            {
                logger.error("Failed to transfer all hints to {}: {}", address, hostId);
                throw new RuntimeException("Failed to transfer all hints to " + hostId);
            }
        }

View on GitHub (pinned to 88fd0f6a0e)