{"record":{"id":"949c2b75754d1911","repo":"apache/cassandra","slug":"failed-to-list-directory-files-in-s-inconsistent","errorCode":null,"errorMessage":"Failed to list directory files in %s, inconsistent disk state for transaction %s","messagePattern":"Failed to list directory files in (.+?), inconsistent disk state for transaction (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java","lineNumber":190,"sourceCode":"        if (txnFile.completed())\n        { // if after re-reading the txn is completed then filter accordingly\n            setTemporary(txnFile, oldFiles.values(), newFiles.values());\n            return;\n        }\n\n        logger.error(\"Failed to classify files in {}\\n\" +\n                     \"Some old files are missing but the txn log is still there and not completed\\n\" +\n                     \"Files in folder:\\n{}\\nTxn: {}\",\n                     folder,\n                     files.isEmpty()\n                        ? \"\\t-\"\n                        : String.join(\"\\n\", files.keySet().stream().map(f -> String.format(\"\\t%s\", f)).collect(Collectors.toList())),\n                     txnFile.toString(true));\n\n        // some old files are missing and yet the txn is still there and not completed\n        // something must be wrong (see comment at the top of LogTransaction requiring txn to be\n        // completed before obsoleting or aborting sstables)\n        throw new RuntimeException(String.format(\"Failed to list directory files in %s, inconsistent disk state for transaction %s\",\n                                                 folder,\n                                                 txnFile));\n    }\n\n    /** See if all files are present */\n    private static boolean allFilesPresent(Map<LogRecord, Set<File>> oldFiles)\n    {\n        return !oldFiles.entrySet().stream()\n                        .filter((e) -> e.getKey().numFiles > e.getValue().size())\n                        .findFirst().isPresent();\n    }\n\n    private void setTemporary(LogFile txnFile, Collection<Set<File>> oldFiles, Collection<Set<File>> newFiles)\n    {\n        Collection<Set<File>> temporary = txnFile.committed() ? oldFiles : newFiles;\n        temporary.stream()\n                 .flatMap(Set::stream)\n                 .forEach((f) -> this.files.put(f, FileType.TEMPORARY));","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java#L172-L208","documentation":"After replaying a transaction log, the lister classifies files as 'old' (pre-transaction) or 'new' (added by the transaction). If the transaction is still present (not completed) yet some recorded old files are missing on disk, the invariant 'transactions must be completed before obsoleting/aborting sstables' is broken, and this RuntimeException is thrown reporting inconsistent disk state.","triggerScenarios":"During startup file listing when a txn_compaction log shows a non-completed transaction whose REMOVE-listed sstable files are absent from the directory.","commonSituations":"Deleting sstable files by hand while a transaction was in progress; interrupted compaction where data files were manually removed; backup/restore tooling copying only sstables and not matching the txn log state; disk cleanup scripts removing 'temporary' files.","solutions":["Compare the exception's printed file list against the transaction log contents to identify missing files","Restore the missing sstables from backup or another replica","If the transaction log is stale/orphaned and the operation is safe, remove the incomplete txn_compaction log (after backup) so replay skips it","Never manually delete sstables or .log files while Cassandra is running — use nodetool to drop/compact","Rebuild the node (repair or bootstrap) if the disk state cannot be reconciled"],"exampleFix":"// before: manual cleanup removed sstables recorded by txn log\nrm /var/lib/cassandra/data/ks/cf/*-Data.db\n// after: let Cassandra manage files\nnodetool compact ks cf   # or restore missing sstables from backup","handlingStrategy":"validation","validationCode":"// before starting a node against restored data, verify txn log completeness\nfor each txn_compaction_*.log: check all files recorded in ADD/REMOVE records exist on disk","typeGuard":null,"tryCatchPattern":"try {\n    lister.list();\n} catch (RuntimeException e) {\n    if (String.valueOf(e.getMessage()).contains(\"inconsistent disk state for transaction\")) {\n        logger.error(\"Txn {} references missing files — restore from backup or rebuild node\", e);\n    } else throw e;\n}","preventionTips":["Never hand-delete sstables or 'temporary' files from data directories","Use nodetool DROP/compact instead of filesystem cleanup","Restore backups as complete directory snapshots including txn logs","Reconcile restored nodes with repair (nodetool repair) after any manual intervention"],"tags":["sstable","transaction-log","disk-state"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}