apache/cassandra · error · RuntimeException

Failed to list directory files in

Error message

Failed to list directory files in %s, inconsistent disk state for transaction %s

What it means

After replaying a transaction log, the lister classifies files as 'old' (pre-transaction) or 'new' (added by the transaction). If the transaction is still present (not completed) yet some recorded old files are missing on disk, the invariant 'transactions must be completed before obsoleting/aborting sstables' is broken, and this RuntimeException is thrown reporting inconsistent disk state.

Solutions

  1. Compare the exception's printed file list against the transaction log contents to identify missing files
  2. Restore the missing sstables from backup or another replica
  3. If the transaction log is stale/orphaned and the operation is safe, remove the incomplete txn_compaction log (after backup) so replay skips it
  4. Never manually delete sstables or .log files while Cassandra is running — use nodetool to drop/compact
  5. Rebuild the node (repair or bootstrap) if the disk state cannot be reconciled

Example fix

// before: manual cleanup removed sstables recorded by txn log
rm /var/lib/cassandra/data/ks/cf/*-Data.db
// after: let Cassandra manage files
nodetool compact ks cf   # or restore missing sstables from backup
Defensive patterns

Strategy: validation

Validate before calling

// before starting a node against restored data, verify txn log completeness
for each txn_compaction_*.log: check all files recorded in ADD/REMOVE records exist on disk

Try / catch

try {
    lister.list();
} catch (RuntimeException e) {
    if (String.valueOf(e.getMessage()).contains("inconsistent disk state for transaction")) {
        logger.error("Txn {} references missing files — restore from backup or rebuild node", e);
    } else throw e;
}

Prevention

When it happens

Trigger: During startup file listing when a txn_compaction log shows a non-completed transaction whose REMOVE-listed sstable files are absent from the directory.

Common situations: Deleting sstable files by hand while a transaction was in progress; interrupted compaction where data files were manually removed; backup/restore tooling copying only sstables and not matching the txn log state; disk cleanup scripts removing 'temporary' files.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/949c2b75754d1911. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java:190

        if (txnFile.completed())
        { // if after re-reading the txn is completed then filter accordingly
            setTemporary(txnFile, oldFiles.values(), newFiles.values());
            return;
        }

        logger.error("Failed to classify files in {}\n" +
                     "Some old files are missing but the txn log is still there and not completed\n" +
                     "Files in folder:\n{}\nTxn: {}",
                     folder,
                     files.isEmpty()
                        ? "\t-"
                        : String.join("\n", files.keySet().stream().map(f -> String.format("\t%s", f)).collect(Collectors.toList())),
                     txnFile.toString(true));

        // some old files are missing and yet the txn is still there and not completed
        // something must be wrong (see comment at the top of LogTransaction requiring txn to be
        // completed before obsoleting or aborting sstables)
        throw new RuntimeException(String.format("Failed to list directory files in %s, inconsistent disk state for transaction %s",
                                                 folder,
                                                 txnFile));
    }

    /** See if all files are present */
    private static boolean allFilesPresent(Map<LogRecord, Set<File>> oldFiles)
    {
        return !oldFiles.entrySet().stream()
                        .filter((e) -> e.getKey().numFiles > e.getValue().size())
                        .findFirst().isPresent();
    }

    private void setTemporary(LogFile txnFile, Collection<Set<File>> oldFiles, Collection<Set<File>> newFiles)
    {
        Collection<Set<File>> temporary = txnFile.committed() ? oldFiles : newFiles;
        temporary.stream()
                 .flatMap(Set::stream)
                 .forEach((f) -> this.files.put(f, FileType.TEMPORARY));

View on GitHub (pinned to 88fd0f6a0e)