{"record":{"id":"b776a957ab4ac788","repo":"neondatabase/neon","slug":"reading-mtime","errorCode":null,"errorMessage":"Reading mtime","messagePattern":"Reading mtime","errorType":"exception","errorClass":"DownloadError","httpStatus":null,"severity":"error","filePath":"libs/remote_storage/src/local_fs.rs","lineNumber":555,"sourceCode":"                take = end - start;\n            }\n        }\n\n        let source = ReaderStream::new(file.take(take));\n\n        let metadata = self\n            .read_storage_metadata(&target_path)\n            .await\n            .map_err(DownloadError::Other)?;\n\n        let cancel_or_timeout = crate::support::cancel_or_timeout(self.timeout, cancel.clone());\n        let source = crate::support::DownloadStream::new(cancel_or_timeout, source);\n\n        Ok(Download {\n            metadata,\n            last_modified: file_metadata\n                .modified()\n                .map_err(|e| DownloadError::Other(anyhow::anyhow!(e).context(\"Reading mtime\")))?,\n            etag,\n            download_stream: Box::pin(source),\n        })\n    }\n\n    async fn delete(&self, path: &RemotePath, _cancel: &CancellationToken) -> anyhow::Result<()> {\n        let file_path = path.with_base(&self.storage_root);\n        match fs::remove_file(&file_path).await {\n            Ok(()) => Ok(()),\n            // The file doesn't exist. This shouldn't yield an error to mirror S3's behaviour.\n            // See https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html\n            // > If there isn't a null version, Amazon S3 does not remove any objects but will still respond that the command was successful.\n            Err(e) if e.kind() == ErrorKind::NotFound => Ok(()),\n            Err(e) => Err(anyhow::anyhow!(e)),\n        }\n    }\n\n    async fn delete_objects(","sourceCodeStart":537,"sourceCodeEnd":573,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/remote_storage/src/local_fs.rs#L537-L573","documentation":"While constructing a Download from LocalFileSystem, std's file_metadata.modified() failed and the io::Error is wrapped with the context 'Reading mtime'. The file was opened moments earlier, so this indicates a race (the file was deleted or replaced between open and metadata read), a filesystem that cannot supply mtime (some FUSE/network mounts, unusual volume drivers), or a permission/IO error on the stat call.","triggerScenarios":"Concurrent delete or rename of the file between open/stat and the modified() call; FUSE or network filesystems returning EPERM/EINVAL for mtime; permission changes on the file mid-download.","commonSituations":"GC or cleanup jobs racing active downloads; test harnesses wiping the workspace concurrently; containers with exotic volume drivers that do not implement all stat fields.","solutions":["Retry the download — if the file was deleted, the retry surfaces a clean NotFound instead","Ensure no concurrent process deletes files under the storage root during downloads","If on FUSE/exotic filesystems, verify `stat <file>` works from a shell on the same host","Inspect the chained io::Error kind in the context chain to distinguish deletion races (ENOENT) from filesystem limitations (EPERM/EINVAL)"],"exampleFix":"// before: single stat, races with concurrent deletes\nlet last_modified = file_metadata.modified()\n    .map_err(|e| DownloadError::Other(anyhow::anyhow!(e).context(\"Reading mtime\")))?;\n\n// after: stat first; a vanished file becomes a clean NotFound\nlet file_metadata = fs::metadata(&file_path).await\n    .map_err(|e| match e.kind() {\n        std::io::ErrorKind::NotFound => DownloadError::NotFound,\n        _ => DownloadError::Other(e.into()),\n    })?;\nlet last_modified = file_metadata.modified()\n    .map_err(|e| DownloadError::Other(anyhow::anyhow!(e).context(\"Reading mtime\")))?;","handlingStrategy":"retry","validationCode":"// Check the file is still present before requesting the download (narrows the race window).\nasync fn file_still_there(storage_root: &Utf8Path, path: &RemotePath) -> bool {\n    tokio::fs::try_exists(path.with_base(storage_root)).await.unwrap_or(false)\n}","typeGuard":null,"tryCatchPattern":"// Race-prone stat: retry once; a deleted file then surfaces as NotFound.\nmatch local_fs.download(&from, &cancel).await {\n    Ok(dl) => Ok(dl),\n    Err(DownloadError::Other(e)) if format!(\"{e:#}\").contains(\"Reading mtime\") => {\n        tracing::warn!(\"mtime read raced for {from}; retrying\");\n        tokio::time::sleep(Duration::from_millis(100)).await;\n        local_fs.download(&from, &cancel).await\n    }\n    Err(e) => Err(e),\n}","preventionTips":["Do not delete files under the storage root while downloads are active — coordinate with a lock or tombstone scheme","On exotic mounts (FUSE/network volumes), verify `stat` returns mtime from the same host before deploying","Read the chained io::Error kind: ENOENT means a deletion race; EPERM/EINVAL points at the filesystem"],"tags":["local-filesystem","file-metadata","race-condition","download","mtime"],"backgroundTag":"file-metadata-unavailable","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}