{"record":{"id":"5730cea79f79169c","repo":"apache/cassandra","slug":"cleaning-up-orphaned-preinitialize-at-epoch-r","errorCode":null,"errorMessage":"Cleaning up orphaned PreInitialize at epoch {} - restarting in gossip mode","messagePattern":"Cleaning up orphaned PreInitialize at epoch (.+?) - restarting in gossip mode","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/tcm/log/SystemKeyspaceStorage.java","lineNumber":103,"sourceCode":"            logger.error(\"Could not persist the entry {} proceeding with in-memory commit.\", entry, t);\n        }\n    }\n\n    public synchronized static boolean hasAnyEpoch()\n    {\n        String query = String.format(\"SELECT epoch, kind FROM %s.%s LIMIT 2\", SchemaConstants.SYSTEM_KEYSPACE_NAME, NAME);\n\n        int count = 0;\n        long preInitializeEpoch = -1;\n        for (UntypedResultSet.Row row : executeInternal(query))\n        {\n            count++;\n            if (Transformation.Kind.fromId(row.getInt(\"kind\")) == Transformation.Kind.PRE_INITIALIZE_CMS)\n                preInitializeEpoch = row.getLong(\"epoch\");\n        }\n        if (count == 1 && preInitializeEpoch != -1)\n        {\n            logger.warn(\"Cleaning up orphaned PreInitialize at epoch {} - restarting in gossip mode\", preInitializeEpoch);\n            String cleanupQuery = String.format(\"DELETE from %s.%s where epoch = %d\", SchemaConstants.SYSTEM_KEYSPACE_NAME, NAME, preInitializeEpoch);\n            executeInternal(cleanupQuery);\n            return false;\n        }\n\n        return count > 0;\n    }\n\n    @Override\n    public MetadataSnapshots snapshots()\n    {\n        return snapshots.get();\n    }\n\n    public void truncate()\n    {\n        Keyspace.open(SchemaConstants.SYSTEM_KEYSPACE_NAME).getColumnFamilyStore(NAME).truncateBlockingWithoutSnapshot();\n    }","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/log/SystemKeyspaceStorage.java#L85-L121","documentation":"SystemKeyspaceStorage.hasAnyEpoch reads the local cluster_metadata log table; if the log contains exactly one entry and it is a PRE_INITIALIZE_CMS transformation, that PreInitialize is orphaned (its node never completed CMS initialization). The method logs a warning, deletes that row, and returns false so startup proceeds in gossip mode rather than TCM mode.","triggerScenarios":"Node startup calls hasAnyEpoch when the local log holds only a lone PRE_INITIALIZE_CMS row at some epoch - produced when a previous bootstrap created the pre-initialize entry but crashed or was interrupted before registering the CMS; the cleanup DELETE runs and startup falls back to gossip.","commonSituations":"Node crashed mid-bootstrap during first CMS initialization; aborted cluster bring-up where seeds were later re-seeded; stale test/node data directories reused after a failed TCM bootstrap.","solutions":["No operator action usually needed: the orphaned row is deleted automatically and gossip mode resumes.","Verify the node then bootstraps normally and registers/joins the CMS; check logs for subsequent epoch writes.","If it recurs, investigate why bootstrap keeps dying between pre-initialize and CMS registration (check earlier stack traces).","Ensure the local system keyspace data is not being reused from an unrelated cluster."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Detect orphaned PreInitialize before startup decisions\nlong count = countEpochRows();\nlong preInit = findPreInitializeEpoch();\nif (count == 1 && preInit != -1) {\n    logger.info(\"Orphaned PreInitialize at {} will be cleaned up; gossip mode expected\", preInit);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Avoid killing nodes during first-time CMS bootstrap.","Do not reuse node data directories across clusters/tests.","Confirm bootstrap completes (CMS registered) before assuming TCM mode is active."],"tags":["tcm","bootstrap","system-keyspace","pre-initialize"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}