{"record":{"id":"5e745100d4f00a0a","repo":"apache/cassandra","slug":"failed-to-remove-hard-link-files","errorCode":null,"errorMessage":"Failed to remove hard link files","messagePattern":"Failed to remove hard link files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/db/streaming/ComponentContext.java","lineNumber":96,"sourceCode":"        @SuppressWarnings(\"resource\") // file channel will be closed by Caller\n        FileChannel channel = toTransfer.newReadChannel();\n\n        assert size == channel.size() : String.format(\"Entire sstable streaming expects %s file size to be %s but got %s.\",\n                                                      component, size, channel.size());\n        return channel;\n    }\n\n    @Override\n    public void close()\n    {\n        Throwable accumulate = null;\n        for (File file : hardLinks.values())\n            accumulate = FileUtils.deleteWithConfirm(file, accumulate);\n\n        hardLinks.clear();\n\n        if (accumulate != null)\n            logger.warn(\"Failed to remove hard link files\", accumulate);\n    }\n}\n","sourceCodeStart":78,"sourceCodeEnd":99,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/db/streaming/ComponentContext.java#L78-L99","documentation":"ComponentContext manages hard links created for streamed sstable components and deletes them on close. deleteWithConfirm accumulates any failures; if the accumulated exception is non-null, close logs 'Failed to remove hard link files' as a warning. The links are typically already superseded; this signals leftover files consuming disk until manually removed.","triggerScenarios":"close() is called (stream writer aborted or completed) and one or more hard-linked files cannot be deleted — usually because another process holds them open, or a permission/IO error prevents removal.","commonSituations":"Files still open by a concurrent reader at close time; filesystem permission problems; NFS/EFS quirkiness where unlink of an open file behaves unexpectedly; disk errors.","solutions":["Manually delete leftover hard-link files from the streamed sstable directories after confirming no process uses them","Check directory permissions for the Cassandra user on the data directories","Look at the accumulated cause in the log to identify which file failed and why","If on NFS/EBS-backed storage, verify the filesystem supports unlink semantics Cassandra expects"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// ensure Cassandra user owns data dirs and no stale processes hold files\n// ls -l on hardlink dirs; lsof +D <data_dir> before manual cleanup","typeGuard":null,"tryCatchPattern":"try (ComponentContext ctx = new ComponentContext(...)) {\n    // use component\n} // close() logs warning on delete failure; clean leftover files manually afterward","preventionTips":["Keep Cassandra data directories on local POSIX filesystems rather than NFS","Ensure consistent ownership/permissions for the cassandra user","After aborts, sweep for leftover hard-link files","Watch disk usage so orphans don't accumulate"],"tags":["filesystem","cleanup","hardlink","io"],"backgroundTag":"file-delete-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}