apache/cassandra · critical · org.apache.cassandra.io.util.StartupException

ERR_WRONG_DISK_STATE

ERR_WRONG_DISK_STATE

Error message

Cannot remove temporary or obsoleted files for %s due to a problem with transaction log files. Please check records with problems in the log messages above and fix them. Refer to the 3.0 upgrading instructions in NEWS.txt for a description of transaction log files.

What it means

During table initialization (after restart), Cassandra removes temporary and obsoleted files left behind by unfinished compactions/flushes using the transaction log (sstable activity) files. If LifecycleTransaction.removeUnfinishedLeftovers reports records with problems, startup of the table is aborted with a StartupException (ERR_WRONG_DISK_STATE) so the operator can fix disk state manually rather than silently deleting or losing data.

Source

Thrown at src/java/org/apache/cassandra/db/ColumnFamilyStore.java:833

        return new ColumnFamilyStore(keyspace, columnFamily,
                                     directories.getUIDGenerator(SSTableIdFactory.instance.defaultBuilder()),
                                     metadata, directories, loadSSTables, registerBookkeeping, addIndexes);
    }

    /**
     * Removes unnecessary files from the cf directory at startup: these include temp files, orphans, zero-length files
     * and compacted sstables. Files that cannot be recognized will be ignored.
     */
    public static void scrubDataDirectories(TableMetadata metadata) throws StartupException
    {
        Directories directories = new Directories(metadata);
        Set<File> cleanedDirectories = new HashSet<>();

        directories.removeTemporaryDirectories();

        logger.trace("Removing temporary or obsoleted files from unfinished operations for table {}", metadata.name);
        if (!LifecycleTransaction.removeUnfinishedLeftovers(metadata))
            throw new StartupException(StartupException.ERR_WRONG_DISK_STATE,
                                       String.format("Cannot remove temporary or obsoleted files for %s due to a problem with transaction " +
                                                     "log files. Please check records with problems in the log messages above and fix them. " +
                                                     "Refer to the 3.0 upgrading instructions in NEWS.txt " +
                                                     "for a description of transaction log files.", metadata));

        logger.trace("Further extra check for orphan sstable files for {}", metadata.name);
        for (Map.Entry<Descriptor,Set<Component>> sstableFiles : directories.sstableLister(Directories.OnTxnErr.IGNORE).list().entrySet())
        {
            Descriptor desc = sstableFiles.getKey();
            File directory = desc.directory;
            Set<Component> components = sstableFiles.getValue();

            if (!cleanedDirectories.contains(directory))
            {
                cleanedDirectories.add(directory);
                for (File tmpFile : desc.getTemporaryFiles())
                {
                    logger.info("Removing unfinished temporary file {}", tmpFile);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Read the 'records with problems' log messages above the exception and manually reconcile the listed files (delete or complete the transaction as guided)
2. Follow the 3.0 upgrading instructions in NEWS.txt for transaction log file handling
3. Remove orphaned sstable files together with their matching *_txn leftovers per the recorded transactions; when in doubt remove only the obsolete temp files listed and retry startup
4. Restore the table's data directory from a consistent backup/snapshot, then start again

Example fix

// before
// startup fails with ERR_WRONG_DISK_STATE; data dir has stale sstable without matching txn records
// after
# remove the stale sstable + its *_txn leftover per NEWS.txt 3.0 instructions, then:
nodetool drain && systemctl restart cassandra  -> startup succeeds

// StartupException cannot be caught in-process; treat as operator alert
try { startNode(); } catch (StartupException e) { if (e.errorCode == StartupException.ERR_WRONG_DISK_STATE) { pageOnCall("reconcile txn log leftovers per NEWS.txt"); } }

When it happens

Trigger: Node restart/startup for a table whose transaction log files reference files that are missing, still present unexpectedly, or otherwise inconsistent - detected in ColumnFamilyStore initialization after directories.removeTemporaryDirectories().

Common situations: Interrupted upgrades from pre-3.0, manually copying/deleting sstable files while transaction logs remain, restore/snapshot operations that moved sstables without their _txn companions, disk failures or partial rsync of data dirs.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Configured ${configName} "${intf}" could not be found
• Configured ${configName} "${intf}" was found, but had no add
• Missing required directive CommitLogSync
• memtable_cleanup_threshold must be >= 0.01, but was ${conf.m
• memtable_cleanup_threshold must be <= 0.99, but was ${conf.m

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/98ef2036644cf173. Report an issue: GitHub.