{"record":{"id":"0a6ca0a670366156","repo":"apache/cassandra","slug":"fsync-s-failed-errno-s-s","errorCode":null,"errorMessage":"fsync(%s) failed, errno (%s) %s","messagePattern":"fsync\\((.+?)\\) failed, errno \\((.+?)\\) (.+?)","errorType":"exception","errorClass":"FSWriteError","httpStatus":null,"severity":"critical","filePath":"src/java/org/apache/cassandra/utils/NativeLibrary.java","lineNumber":347,"sourceCode":"\n        try\n        {\n            wrappedLibrary.callFsync(fd);\n        }\n        catch (UnsatisfiedLinkError e)\n        {\n            // JNA is unavailable just skipping Direct I/O\n        }\n        catch (RuntimeException e)\n        {\n            if (!(e instanceof LastErrorException))\n                throw e;\n\n            if (REQUIRE)\n            {\n                String errMsg = String.format(\"fsync(%s) failed, errno (%s) %s\", fd, errno(e), e.getMessage());\n                logger.warn(errMsg);\n                throw new FSWriteError(e, errMsg);\n            }\n        }\n    }\n\n    public static void tryCloseFD(int fd)\n    {\n        if (fd == -1)\n            return;\n\n        try\n        {\n            wrappedLibrary.callClose(fd);\n        }\n        catch (UnsatisfiedLinkError e)\n        {\n            // JNA is unavailable just skipping Direct I/O\n        }\n        catch (RuntimeException e)","sourceCodeStart":329,"sourceCodeEnd":365,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/utils/NativeLibrary.java#L329-L365","documentation":"NativeLibrary.trySync fsyncs a file descriptor (when native fsync is available). If fsync fails and the REQUIRE flag is set, it logs and throws FSWriteError carrying the formatted message with fd, errno and the OS error text. This means the OS could not flush the file's dirty pages to stable storage.","triggerScenarios":"Calling NativeLibrary.trySync(fd) (directly or via commitlog/sstable sync paths) when the underlying file descriptor is bad, the file was deleted/unmounted, the disk is full or failing, or I/O errors occur on the device.","commonSituations":"Disk hardware failure or removal, filesystem unmounted or remounted read-only (ENOSPC, EIO, EBADF), container environments where the volume is unstable, or closing files concurrently while another thread syncs them.","solutions":["Read errno and message in the FSWriteError: ENOSPC → free disk space; EIO → check dmesg/smartctl for disk failure; EBADF → look for premature fd close","Replace or repair the failing disk / restore the filesystem and restart the node","Ensure the data and commitlog directories are on healthy, writable mounts","Check for concurrent close/sync races in code managing the fd; run nodetool repair afterwards to verify data integrity"],"exampleFix":"// before (diagnostic)\n// FSWriteError: fsync(35) failed, errno (28) No space left on device\n// after\ncf_du_out> df -h /var/lib/cassandra/commitlog  # free space or add disk, then restart node","handlingStrategy":"retry","validationCode":"// preflight: ensure the mount is writable and has free space\nProcess p = Runtime.getRuntime().exec(new String[]{\"sh\",\"-c\",\"df -h /var/lib/cassandra/commitlog\"});","typeGuard":null,"tryCatchPattern":"try { NativeLibrary.trySync(fd); } catch (FSWriteError e) { switch (errno(e)) { case ENOSPC: freeDiskSpace(); break; case EIO: pageOpsTeamAndReplaceDisk(); break; default: restartNode(); } }","preventionTips":["Monitor disk space and I/O errors (node_exporter, dmesg)","Keep commitlog and data on healthy, redundant storage","Avoid sharing fd ownership across threads; sync before close","Set up alerting on FSWriteError log patterns"],"tags":["fsync","disk-io","fs-write-error","durability"],"backgroundTag":"file-write-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}