What causes "Missing ETag header" in neondatabase/neon?

Point dev/test at an endpoint that returns ETag (real S3 or MinIO)

neondatabase/neon · error · DownloadError

Missing ETag header

Error message

Missing ETag header

What it means

Thrown by S3Bucket's download path after a successful GetObject when the S3 response carries no ETag header. Real AWS S3 (and compliant servers like MinIO) always set ETag on GetObject, so this almost always means the configured endpoint is a partially S3-compatible emulator or a proxy that strips headers. The ETag is required because the returned Download struct exposes etag for later conditional requests (if_none_match).

Source

Thrown at libs/remote_storage/src/s3_bucket.rs:318

                    kind,
                    AttemptOutcome::Err,
                    started_at,
                );

                return Err(DownloadError::Other(
                    anyhow::Error::new(e).context("download s3 object"),
                ));
            }
        };

        // even if we would have no timeout left, continue anyways. the caller can decide to ignore
        // the errors considering timeouts and cancellation.
        let remaining = self.timeout.saturating_sub(started_at.elapsed());

        let metadata = object_output.metadata().cloned().map(StorageMetadata);
        let etag = object_output
            .e_tag
            .ok_or(DownloadError::Other(anyhow::anyhow!("Missing ETag header")))?
            .into();
        let last_modified = object_output
            .last_modified
            .ok_or(DownloadError::Other(anyhow::anyhow!(
                "Missing LastModified header"
            )))?
            .try_into()
            .map_err(|e: ConversionError| DownloadError::Other(e.into()))?;

        let body = object_output.body;
        let body = ByteStreamAsStream::from(body);
        let body = PermitCarrying::new(permit, body);
        let body = TimedDownload::new(started_at, body);

        let cancel_or_timeout = crate::support::cancel_or_timeout(remaining, cancel.clone());
        let body = crate::support::DownloadStream::new(cancel_or_timeout, body);

        Ok(Download {

View on GitHub (pinned to 8f60b04da4)

Solutions

Point dev/test at an endpoint that returns ETag (real S3 or MinIO)
Verify the endpoint directly: aws s3api get-object --bucket <b> --key <k> /dev/null --query ETag
Fix or upgrade the emulator/gateway to include ETag in GetObject responses
If you control the server, set the ETag header on all GetObject responses

Example fix

// before: dev/test against an emulator that omits ETag
remote_storage_config = { bucket = "local", endpoint = "http://mock:5000" }
// after: use a compliant S3 server
remote_storage_config = { bucket = "local", endpoint = "http://minio:9000" }

Defensive patterns

Strategy: try-catch

Validate before calling

use aws_sdk_s3::Client;

async fn endpoint_returns_etag(client: &Client, bucket: &str, key: &str) -> anyhow::Result<bool> {
    let head = client.head_object().bucket(bucket).key(key).send().await?;
    Ok(head.e_tag.is_some())
}

Type guard

fn is_missing_etag(err: &remote_storage::DownloadError) -> bool {
    matches!(err, remote_storage::DownloadError::Other(e)
        if e.to_string().contains("Missing ETag header"))
}

Try / catch

match storage.download(&path, &cancel).await {
    Err(DownloadError::Other(e)) if e.to_string().contains("Missing ETag header") => {
        // endpoint incompatibility: do not retry; report the endpoint configuration
    }
    Err(DownloadError::NotFound) => { /* object absent */ }
    other => other?,
}

Prevention

Use real S3 or MinIO in integration tests
Add a startup probe that HEADs a known object and asserts ETag presence
Alert on this exact message: it signals protocol non-compliance, not a transient failure

When it happens

Trigger: Calling GenericRemoteStorage::download / download_object (e.g. a pageserver fetching a layer or timeline index) against an S3-compatible endpoint whose GetObject response omits the ETag header, or through a proxy that removes response headers.

Common situations: Local dev/CI against a hand-rolled mock or partially compliant S3 emulator; a corporate proxy or gateway stripping headers; an S3-like provider that skips ETag for some object classes.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/89d4abed8a2afa32. Report an issue: GitHub.