neondatabase/neon · error · DownloadError

head_object doesn't contain last_modified or content_length

Error message

head_object doesn't contain last_modified or content_length

What it means

In the HEAD-based per-key listing helper, the HeadObject response must carry both last_modified and content_length to construct a ListingObject. If either is missing, the request technically succeeded but the endpoint returned an incomplete HeadObject response, which real S3 never does.

Source

Thrown at libs/remote_storage/src/s3_bucket.rs:823

                );
                return Err(DownloadError::NotFound);
            }
            Err(e) => {
                crate::metrics::BUCKET_METRICS.req_seconds.observe_elapsed(
                    kind,
                    AttemptOutcome::Err,
                    started_at,
                );

                return Err(DownloadError::Other(
                    anyhow::Error::new(e).context("s3 head object"),
                ));
            }
        };

        let (Some(last_modified), Some(size)) = (data.last_modified, data.content_length) else {
            return Err(DownloadError::Other(anyhow!(
                "head_object doesn't contain last_modified or content_length"
            )))?;
        };
        Ok(ListingObject {
            key: key.to_owned(),
            last_modified: SystemTime::try_from(last_modified).map_err(|e| {
                DownloadError::Other(anyhow!("can't convert time '{last_modified}': {e}"))
            })?,
            size: size as u64,
        })
    }

    async fn upload(
        &self,
        from: impl Stream<Item = std::io::Result<Bytes>> + Send + Sync + 'static,
        from_size_bytes: usize,
        to: &RemotePath,
        metadata: Option<StorageMetadata>,
        cancel: &CancellationToken,

View on GitHub (pinned to 8f60b04da4)

Solutions

  1. Verify the endpoint returns both fields: aws s3api head-object --bucket <b> --key <k>
  2. Switch dev/CI to real S3 or MinIO
  3. Fix the emulator to set Last-Modified and Content-Length on HeadObject responses

Example fix

# before: incomplete HEAD from emulator
$ aws s3api head-object --bucket b --key k --endpoint-url http://emulator:5000
{}
# after: compliant endpoint
$ aws s3api head-object --bucket b --key k --endpoint-url http://minio:9000
{"LastModified": "...", "ContentLength": 1024, ...}
Defensive patterns

Strategy: try-catch

Validate before calling

use aws_sdk_s3::Client;

async fn head_is_complete(client: &Client, bucket: &str, key: &str) -> anyhow::Result<bool> {
    let head = client.head_object().bucket(bucket).key(key).send().await?;
    Ok(head.last_modified.is_some() && head.content_length.is_some())
}

Type guard

fn is_incomplete_head_response(err: &remote_storage::DownloadError) -> bool {
    matches!(err, remote_storage::DownloadError::Other(e)
        if e.to_string().contains("head_object doesn't contain"))
}

Try / catch

match storage.list_files(&prefix, mode, &cancel).await {
    Err(DownloadError::Other(e)) if e.to_string().contains("head_object doesn't contain") => {
        // emulator non-compliance: report endpoint, no retry
    }
    other => other?,
}

Prevention

When it happens

Trigger: Listing objects via the HeadObject path (list_files / listing flows that probe keys) against an S3-compatible emulator that omits Last-Modified or Content-Length in HeadObject responses.

Common situations: CI/dev with mocked S3 servers; gateways that synthesize HEAD responses from metadata-less storage without standard headers.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/82f0b3e3052b3edc. Report an issue: GitHub.