apache/cassandra · error · StartupException

3

3

Error message

Unable to verify sstable files on disk

What it means

During the sstable-verification startup check, Cassandra walks each data directory tree to validate sstable files on disk. If the filesystem walk itself fails (an IOException from Files.walkFileTree, e.g. I/O error or traversal failure), a StartupException with code 3 is thrown wrapping the cause. This indicates Cassandra could not even complete its scan of sstable files, not that specific files are bad.

Source

Thrown at src/java/org/apache/cassandra/service/StartupChecks.java:1026

                    String name = dir.getFileName().toString();
                    return (name.equals(Directories.SNAPSHOT_SUBDIR)
                            || name.equals(Directories.BACKUPS_SUBDIR)
                            || nonSSTablePaths.contains(PathUtils.toCanonicalPath(dir).toString()))
                           ? FileVisitResult.SKIP_SUBTREE
                           : FileVisitResult.CONTINUE;
                }
            };

            for (String dataDir : DatabaseDescriptor.getAllDataFileLocations())
            {
                try
                {
                    Files.walkFileTree(new File(dataDir).toPath(), sstableVisitor);
                }
                catch (IOException e)
                {
                    throw new StartupException(3, "Unable to verify sstable files on disk", e);
                }
            }

            if (!invalid.isEmpty())
                throw new StartupException(StartupException.ERR_WRONG_DISK_STATE,
                                           String.format("Detected unreadable sstables %s, please check " +
                                                         "NEWS.txt and ensure that you have upgraded through " +
                                                         "all required intermediate versions, running " +
                                                         "upgradesstables",
                                                         Joiner.on(",").join(invalid)));

            if (!withIllegalGenId.isEmpty())
                throw new StartupException(StartupException.ERR_WRONG_CONFIG,
                                           "UUID sstable identifiers are disabled but some sstables have been " +
                                           "created with UUID identifiers. You have to either delete those " +
                                           "sstables or enable UUID based sstable identifers in cassandra.yaml " +
                                           "(uuid_sstable_identifiers_enabled). The list of affected sstables is: " +
                                           Joiner.on(", ").join(withIllegalGenId) + ". If you decide to delete sstables, " +

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Read the wrapped cause (getCause) in the startup log to identify which path/IOException aborted the walk and fix that specific filesystem issue.
2. Repair directory permissions inside the data directories (chown/chmod) and clear broken symlinks.
3. Raise the open-file limit (ulimit -n / systemd LimitNOFILE) if the walk hit EMFILE.

Example fix

# before: data sub-dir unreadable by cassandra user
sudo chmod -R u+rwx /var/lib/cassandra/data
sudo chown -R cassandra:cassandra /var/lib/cassandra/data
# after: walkFileTree completes and startup proceeds

// pre-flight: ensure the walk is possible (read+execute on data tree, sane ulimit)
find /var/lib/cassandra/data -type d ! -readable -print   # should print nothing
ulimit -n   # should be comfortably large (e.g. 100000)

try {
    startupChecks.execute(config);
} catch (StartupException e) {
    if (e.getCode() == 3 && e.getCause() instanceof java.io.IOException)
        logger.error("Sstable scan aborted: fix filesystem issue on data dirs", e.getCause());
}

When it happens

Trigger: StartupChecks sstable check executes Files.walkFileTree(new File(dataDir).toPath(), sstableVisitor); any IOException thrown during traversal (permission errors on subdirectories, I/O errors, too many open files) is wrapped as StartupException(3, "Unable to verify sstable files on disk", e).

Common situations: Broken symlinks or subdirectory permission problems inside data dirs; underlying storage I/O failures; ulimit -n too low during deep directory walks; NFS stale handles.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to list files in %s
• Multiple sstable format implementations with the same name %
• Failed to instantiate sstable format '%s'
• Unable check disk space in '%s'. Perhaps the Cassandra user
• ; unable to start server

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/937223748265a7bb. Report an issue: GitHub.