neondatabase/neon · error · DownloadError

Missing LastModified header

Error message

Missing LastModified header

What it means

Companion to the ETag check in S3Bucket's download path: after GetObject succeeds, LastModified is required to build the Download struct (it feeds Download.last_modified). If the response has no LastModified header, the download fails before any body bytes are read. Real S3 always sends it, so the cause is a non-compliant endpoint or header-stripping proxy.

Source

Thrown at libs/remote_storage/src/s3_bucket.rs:323

                return Err(DownloadError::Other(
                    anyhow::Error::new(e).context("download s3 object"),
                ));
            }
        };

        // even if we would have no timeout left, continue anyways. the caller can decide to ignore
        // the errors considering timeouts and cancellation.
        let remaining = self.timeout.saturating_sub(started_at.elapsed());

        let metadata = object_output.metadata().cloned().map(StorageMetadata);
        let etag = object_output
            .e_tag
            .ok_or(DownloadError::Other(anyhow::anyhow!("Missing ETag header")))?
            .into();
        let last_modified = object_output
            .last_modified
            .ok_or(DownloadError::Other(anyhow::anyhow!(
                "Missing LastModified header"
            )))?
            .try_into()
            .map_err(|e: ConversionError| DownloadError::Other(e.into()))?;

        let body = object_output.body;
        let body = ByteStreamAsStream::from(body);
        let body = PermitCarrying::new(permit, body);
        let body = TimedDownload::new(started_at, body);

        let cancel_or_timeout = crate::support::cancel_or_timeout(remaining, cancel.clone());
        let body = crate::support::DownloadStream::new(cancel_or_timeout, body);

        Ok(Download {
            metadata,
            etag,
            last_modified,
            download_stream: Box::pin(body),
        })

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Switch dev/CI to real S3 or MinIO, which always return Last-Modified
2. Verify with aws s3api get-object --query LastModified against the endpoint
3. Fix the emulator to emit a valid Last-Modified (RFC 7231 IMF-fixdate) on GetObject
4. Check intermediate proxies are not stripping standard headers

Example fix

// before: emulator GetObject handler omits headers
resp.body(data);
// after: set required metadata
resp.insert_header("ETag", format!("\"{}\"", etag));
resp.insert_header("Last-Modified", httpdate::fmt_http_date(SystemTime::now()));
resp.body(data);

use aws_sdk_s3::Client;

async fn endpoint_returns_last_modified(client: &Client, bucket: &str, key: &str) -> anyhow::Result<bool> {
    let head = client.head_object().bucket(bucket).key(key).send().await?;
    Ok(head.last_modified.is_some())
}

fn is_missing_last_modified(err: &remote_storage::DownloadError) -> bool {
    matches!(err, remote_storage::DownloadError::Other(e)
        if e.to_string().contains("Missing LastModified header"))
}

match storage.download(&path, &cancel).await {
    Err(DownloadError::Other(e)) if e.to_string().contains("Missing LastModified header") => {
        // endpoint non-compliance: fail the fetch and surface endpoint config, no retry
    }
    other => other?,
}

When it happens

Trigger: Calling GenericRemoteStorage::download / download_object against an S3-compatible emulator or gateway that omits the Last-Modified response header on GetObject.

Common situations: Mock S3 servers in CI that only implement body streaming; gateways that synthesize GetObject responses from metadata-less backends; misconfigured reverse proxies.

Related errors

• Missing ETag header
• head_object doesn't contain last_modified or content_length
• Failed to delete {}/{} objects
• Received truncated ListObjectVersions response for prefix={p
• can't convert time '{last_modified}': {e}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/4f8a4a322fb34ff7. Report an issue: GitHub.