apache/cassandra · error · StartupException

3

3

Error message

Unable to verify sstable files on disk

What it means

During the sstable-verification startup check, Cassandra walks each data directory tree to validate sstable files on disk. If the filesystem walk itself fails (an IOException from Files.walkFileTree, e.g. I/O error or traversal failure), a StartupException with code 3 is thrown wrapping the cause. This indicates Cassandra could not even complete its scan of sstable files, not that specific files are bad.

Solutions

1. Read the wrapped cause (getCause) in the startup log to identify which path/IOException aborted the walk and fix that specific filesystem issue.
2. Repair directory permissions inside the data directories (chown/chmod) and clear broken symlinks.
3. Raise the open-file limit (ulimit -n / systemd LimitNOFILE) if the walk hit EMFILE.

Example fix

# before: data sub-dir unreadable by cassandra user
sudo chmod -R u+rwx /var/lib/cassandra/data
sudo chown -R cassandra:cassandra /var/lib/cassandra/data
# after: walkFileTree completes and startup proceeds

// pre-flight: ensure the walk is possible (read+execute on data tree, sane ulimit)
find /var/lib/cassandra/data -type d ! -readable -print   # should print nothing
ulimit -n   # should be comfortably large (e.g. 100000)

try {
    startupChecks.execute(config);
} catch (StartupException e) {
    if (e.getCode() == 3 && e.getCause() instanceof java.io.IOException)
        logger.error("Sstable scan aborted: fix filesystem issue on data dirs", e.getCause());
}

When it happens

Trigger: StartupChecks sstable check executes Files.walkFileTree(new File(dataDir).toPath(), sstableVisitor); any IOException thrown during traversal (permission errors on subdirectories, I/O errors, too many open files) is wrapped as StartupException(3, "Unable to verify sstable files on disk", e).

Common situations: Broken symlinks or subdirectory permission problems inside data dirs; underlying storage I/O failures; ulimit -n too low during deep directory walks; NFS stale handles.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to list files in
• Corrupted key cache. Failed to deserialize key of key cache…
• Could not perform startup sequence and join cluster
• Directory doesn't exist
• ERR_WRONG_DISK_STATE

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/937223748265a7bb. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/StartupChecks.java:1026

                    String name = dir.getFileName().toString();
                    return (name.equals(Directories.SNAPSHOT_SUBDIR)
                            || name.equals(Directories.BACKUPS_SUBDIR)
                            || nonSSTablePaths.contains(PathUtils.toCanonicalPath(dir).toString()))
                           ? FileVisitResult.SKIP_SUBTREE
                           : FileVisitResult.CONTINUE;
                }
            };

            for (String dataDir : DatabaseDescriptor.getAllDataFileLocations())
            {
                try
                {
                    Files.walkFileTree(new File(dataDir).toPath(), sstableVisitor);
                }
                catch (IOException e)
                {
                    throw new StartupException(3, "Unable to verify sstable files on disk", e);
                }
            }

            if (!invalid.isEmpty())
                throw new StartupException(StartupException.ERR_WRONG_DISK_STATE,
                                           String.format("Detected unreadable sstables %s, please check " +
                                                         "NEWS.txt and ensure that you have upgraded through " +
                                                         "all required intermediate versions, running " +
                                                         "upgradesstables",
                                                         Joiner.on(",").join(invalid)));

            if (!withIllegalGenId.isEmpty())
                throw new StartupException(StartupException.ERR_WRONG_CONFIG,
                                           "UUID sstable identifiers are disabled but some sstables have been " +
                                           "created with UUID identifiers. You have to either delete those " +
                                           "sstables or enable UUID based sstable identifers in cassandra.yaml " +
                                           "(uuid_sstable_identifiers_enabled). The list of affected sstables is: " +
                                           Joiner.on(", ").join(withIllegalGenId) + ". If you decide to delete sstables, " +

View on GitHub (pinned to 88fd0f6a0e)