apache/cassandra · critical · FSWriteError

fsync( ) failed, errno ( )

Error message

fsync(%s) failed, errno (%s) %s

What it means

NativeLibrary.trySync fsyncs a file descriptor (when native fsync is available). If fsync fails and the REQUIRE flag is set, it logs and throws FSWriteError carrying the formatted message with fd, errno and the OS error text. This means the OS could not flush the file's dirty pages to stable storage.

Solutions

1. Read errno and message in the FSWriteError: ENOSPC → free disk space; EIO → check dmesg/smartctl for disk failure; EBADF → look for premature fd close
2. Replace or repair the failing disk / restore the filesystem and restart the node
3. Ensure the data and commitlog directories are on healthy, writable mounts
4. Check for concurrent close/sync races in code managing the fd; run nodetool repair afterwards to verify data integrity

Example fix

// before (diagnostic)
// FSWriteError: fsync(35) failed, errno (28) No space left on device
// after
cf_du_out> df -h /var/lib/cassandra/commitlog  # free space or add disk, then restart node

// preflight: ensure the mount is writable and has free space
Process p = Runtime.getRuntime().exec(new String[]{"sh","-c","df -h /var/lib/cassandra/commitlog"});

try { NativeLibrary.trySync(fd); } catch (FSWriteError e) { switch (errno(e)) { case ENOSPC: freeDiskSpace(); break; case EIO: pageOpsTeamAndReplaceDisk(); break; default: restartNode(); } }

When it happens

Trigger: Calling NativeLibrary.trySync(fd) (directly or via commitlog/sstable sync paths) when the underlying file descriptor is bad, the file was deleted/unmounted, the disk is full or failing, or I/O errors occur on the device.

Common situations: Disk hardware failure or removal, filesystem unmounted or remounted read-only (ENOSPC, EIO, EBADF), container environments where the volume is unstable, or closing files concurrently while another thread syncs them.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Unable to open hint directory using Native library…
• Accord journal is configured in periodic mode, while…
• Caught exception while waiting for memtable flushes during…
• close( ) failed, errno ( ).
• DiskAccessMode ' ' is not supported

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/0a6ca0a670366156. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/utils/NativeLibrary.java:347

        try
        {
            wrappedLibrary.callFsync(fd);
        }
        catch (UnsatisfiedLinkError e)
        {
            // JNA is unavailable just skipping Direct I/O
        }
        catch (RuntimeException e)
        {
            if (!(e instanceof LastErrorException))
                throw e;

            if (REQUIRE)
            {
                String errMsg = String.format("fsync(%s) failed, errno (%s) %s", fd, errno(e), e.getMessage());
                logger.warn(errMsg);
                throw new FSWriteError(e, errMsg);
            }
        }
    }

    public static void tryCloseFD(int fd)
    {
        if (fd == -1)
            return;

        try
        {
            wrappedLibrary.callClose(fd);
        }
        catch (UnsatisfiedLinkError e)
        {
            // JNA is unavailable just skipping Direct I/O
        }
        catch (RuntimeException e)

View on GitHub (pinned to 88fd0f6a0e)