Cleaning up orphaned PreInitialize at epoch

Error message

Cleaning up orphaned PreInitialize at epoch {} - restarting in gossip mode

What it means

SystemKeyspaceStorage.hasAnyEpoch reads the local cluster_metadata log table; if the log contains exactly one entry and it is a PRE_INITIALIZE_CMS transformation, that PreInitialize is orphaned (its node never completed CMS initialization). The method logs a warning, deletes that row, and returns false so startup proceeds in gossip mode rather than TCM mode.

Solutions

No operator action usually needed: the orphaned row is deleted automatically and gossip mode resumes.
Verify the node then bootstraps normally and registers/joins the CMS; check logs for subsequent epoch writes.
If it recurs, investigate why bootstrap keeps dying between pre-initialize and CMS registration (check earlier stack traces).
Ensure the local system keyspace data is not being reused from an unrelated cluster.

Defensive patterns

Strategy: validation

Validate before calling

// Detect orphaned PreInitialize before startup decisions
long count = countEpochRows();
long preInit = findPreInitializeEpoch();
if (count == 1 && preInit != -1) {
    logger.info("Orphaned PreInitialize at {} will be cleaned up; gossip mode expected", preInit);
}

Prevention

Avoid killing nodes during first-time CMS bootstrap.
Do not reuse node data directories across clusters/tests.
Confirm bootstrap completes (CMS registered) before assuming TCM mode is active.

When it happens

Trigger: Node startup calls hasAnyEpoch when the local log holds only a lone PRE_INITIALIZE_CMS row at some epoch - produced when a previous bootstrap created the pre-initialize entry but crashed or was interrupted before registering the CMS; the cleanup DELETE runs and startup falls back to gossip.

Common situations: Node crashed mid-bootstrap during first CMS initialization; aborted cluster bring-up where seeds were later re-seeded; stale test/node data directories reused after a failed TCM bootstrap.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/5730cea79f79169c. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/log/SystemKeyspaceStorage.java:103

            logger.error("Could not persist the entry {} proceeding with in-memory commit.", entry, t);
        }
    }

    public synchronized static boolean hasAnyEpoch()
    {
        String query = String.format("SELECT epoch, kind FROM %s.%s LIMIT 2", SchemaConstants.SYSTEM_KEYSPACE_NAME, NAME);

        int count = 0;
        long preInitializeEpoch = -1;
        for (UntypedResultSet.Row row : executeInternal(query))
        {
            count++;
            if (Transformation.Kind.fromId(row.getInt("kind")) == Transformation.Kind.PRE_INITIALIZE_CMS)
                preInitializeEpoch = row.getLong("epoch");
        }
        if (count == 1 && preInitializeEpoch != -1)
        {
            logger.warn("Cleaning up orphaned PreInitialize at epoch {} - restarting in gossip mode", preInitializeEpoch);
            String cleanupQuery = String.format("DELETE from %s.%s where epoch = %d", SchemaConstants.SYSTEM_KEYSPACE_NAME, NAME, preInitializeEpoch);
            executeInternal(cleanupQuery);
            return false;
        }

        return count > 0;
    }

    @Override
    public MetadataSnapshots snapshots()
    {
        return snapshots.get();
    }

    public void truncate()
    {
        Keyspace.open(SchemaConstants.SYSTEM_KEYSPACE_NAME).getColumnFamilyStore(NAME).truncateBlockingWithoutSnapshot();
    }

View on GitHub (pinned to 88fd0f6a0e)