neondatabase/neon · error · DownloadError

Reading mtime

Error message

Reading mtime

What it means

While constructing a Download from LocalFileSystem, std's file_metadata.modified() failed and the io::Error is wrapped with the context 'Reading mtime'. The file was opened moments earlier, so this indicates a race (the file was deleted or replaced between open and metadata read), a filesystem that cannot supply mtime (some FUSE/network mounts, unusual volume drivers), or a permission/IO error on the stat call.

Source

Thrown at libs/remote_storage/src/local_fs.rs:555

                take = end - start;
            }
        }

        let source = ReaderStream::new(file.take(take));

        let metadata = self
            .read_storage_metadata(&target_path)
            .await
            .map_err(DownloadError::Other)?;

        let cancel_or_timeout = crate::support::cancel_or_timeout(self.timeout, cancel.clone());
        let source = crate::support::DownloadStream::new(cancel_or_timeout, source);

        Ok(Download {
            metadata,
            last_modified: file_metadata
                .modified()
                .map_err(|e| DownloadError::Other(anyhow::anyhow!(e).context("Reading mtime")))?,
            etag,
            download_stream: Box::pin(source),
        })
    }

    async fn delete(&self, path: &RemotePath, _cancel: &CancellationToken) -> anyhow::Result<()> {
        let file_path = path.with_base(&self.storage_root);
        match fs::remove_file(&file_path).await {
            Ok(()) => Ok(()),
            // The file doesn't exist. This shouldn't yield an error to mirror S3's behaviour.
            // See https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html
            // > If there isn't a null version, Amazon S3 does not remove any objects but will still respond that the command was successful.
            Err(e) if e.kind() == ErrorKind::NotFound => Ok(()),
            Err(e) => Err(anyhow::anyhow!(e)),
        }
    }

    async fn delete_objects(

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Retry the download — if the file was deleted, the retry surfaces a clean NotFound instead
2. Ensure no concurrent process deletes files under the storage root during downloads
3. If on FUSE/exotic filesystems, verify `stat <file>` works from a shell on the same host
4. Inspect the chained io::Error kind in the context chain to distinguish deletion races (ENOENT) from filesystem limitations (EPERM/EINVAL)

Example fix

// before: single stat, races with concurrent deletes
let last_modified = file_metadata.modified()
    .map_err(|e| DownloadError::Other(anyhow::anyhow!(e).context("Reading mtime")))?;

// after: stat first; a vanished file becomes a clean NotFound
let file_metadata = fs::metadata(&file_path).await
    .map_err(|e| match e.kind() {
        std::io::ErrorKind::NotFound => DownloadError::NotFound,
        _ => DownloadError::Other(e.into()),
    })?;
let last_modified = file_metadata.modified()
    .map_err(|e| DownloadError::Other(anyhow::anyhow!(e).context("Reading mtime")))?;

// Check the file is still present before requesting the download (narrows the race window).
async fn file_still_there(storage_root: &Utf8Path, path: &RemotePath) -> bool {
    tokio::fs::try_exists(path.with_base(storage_root)).await.unwrap_or(false)
}

// Race-prone stat: retry once; a deleted file then surfaces as NotFound.
match local_fs.download(&from, &cancel).await {
    Ok(dl) => Ok(dl),
    Err(DownloadError::Other(e)) if format!("{e:#}").contains("Reading mtime") => {
        tracing::warn!("mtime read raced for {from}; retrying");
        tokio::time::sleep(Duration::from_millis(100)).await;
        local_fs.download(&from, &cancel).await
    }
    Err(e) => Err(e),
}

When it happens

Trigger: Concurrent delete or rename of the file between open/stat and the modified() call; FUSE or network filesystems returning EPERM/EINVAL for mtime; permission changes on the file mid-download.

Common situations: GC or cleanup jobs racing active downloads; test harnesses wiping the workspace concurrently; containers with exotic volume drivers that do not implement all stat fields.

Related errors

• Cannot refresh compute configuration in state {:?}
• Remote extensions storage is not configured
• error downloading extension {:?}: {:?}
• Azure GET response contained no response body
• GCS GET response contained no response body

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/b776a957ab4ac788. Report an issue: GitHub.