neondatabase/neon · error · DownloadError

can't convert time

Error message

can't convert time '{last_modified}': {e}

What it means

After a successful HeadObject, last_modified (an aws_smithy_types::DateTime) is converted with SystemTime::try_from; the conversion fails when the timestamp is outside SystemTime's range, most commonly a date before the Unix epoch. The offending timestamp is embedded in the message, so you can see exactly what the endpoint reported.

Solutions

1. Compare the timestamp in the error message with aws s3api head-object for the same key
2. Fix the emulator/uploader so Last-Modified is a valid post-epoch RFC 3339/HTTP date
3. Overwrite or delete the malformed object if the endpoint cannot be fixed

Example fix

// before: emulator writes epoch-relative stamps
response.insert_header("Last-Modified", "Wed, 31 Dec 1969 23:59:59 GMT");
// after: emit a real UTC time
response.insert_header("Last-Modified", httpdate::fmt_http_date(std::time::SystemTime::now()));

use aws_sdk_s3::Client;

async fn last_modified_in_range(client: &Client, bucket: &str, key: &str) -> anyhow::Result<bool> {
    let head = client.head_object().bucket(bucket).key(key).send().await?;
    Ok(head.last_modified
        .map(|t| t.secs() >= 0)
        .unwrap_or(false))
}

fn is_time_conversion_error(err: &remote_storage::DownloadError) -> bool {
    matches!(err, remote_storage::DownloadError::Other(e)
        if e.to_string().contains("can't convert time"))
}

match storage.list_files(&prefix, mode, &cancel).await {
    Err(DownloadError::Other(e)) if e.to_string().contains("can't convert time") => {
        // the message embeds the offending timestamp; fix or remove that object/endpoint
    }
    other => other?,
}

When it happens

Trigger: An S3-compatible endpoint reporting a zero, negative, or otherwise out-of-range Last-Modified for a key being listed, e.g. an emulated default of 1970-01-01 minus an offset, or a gateway with broken clock handling.

Common situations: Emulators defaulting timestamps incorrectly; objects written by tooling with timezone math bugs; systems with wrong RTC clocks.

Related errors

• head_object doesn't contain last_modified or content_length
• Missing ETag header
• Missing LastModified header
• Azure GET response contained no response body
• Failed to delete / objects

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/8b4066994bad00cf. Report an issue: GitHub.

Appendix: source

Thrown at libs/remote_storage/src/s3_bucket.rs:829

                    AttemptOutcome::Err,
                    started_at,
                );

                return Err(DownloadError::Other(
                    anyhow::Error::new(e).context("s3 head object"),
                ));
            }
        };

        let (Some(last_modified), Some(size)) = (data.last_modified, data.content_length) else {
            return Err(DownloadError::Other(anyhow!(
                "head_object doesn't contain last_modified or content_length"
            )))?;
        };
        Ok(ListingObject {
            key: key.to_owned(),
            last_modified: SystemTime::try_from(last_modified).map_err(|e| {
                DownloadError::Other(anyhow!("can't convert time '{last_modified}': {e}"))
            })?,
            size: size as u64,
        })
    }

    async fn upload(
        &self,
        from: impl Stream<Item = std::io::Result<Bytes>> + Send + Sync + 'static,
        from_size_bytes: usize,
        to: &RemotePath,
        metadata: Option<StorageMetadata>,
        cancel: &CancellationToken,
    ) -> anyhow::Result<()> {
        let kind = RequestKind::Put;
        let _permit = self.permit(kind, cancel).await?;

        let started_at = start_measuring_requests(kind);

View on GitHub (pinned to 8f60b04da4)