neondatabase/neon · critical

Read back file doesn't match original

Error message

Read back file doesn't match original

What it means

check_storage_permissions (run at endpoint_storage startup unless --no-s3-check-on-startup) writes a uniquely-named probe file to the remote bucket, downloads it back, and compares the strings. A mismatch means the storage backend is not faithfully returning what was written — for example multiple instances sharing one bucket through a caching proxy — and the check bails after logging both bodies.

Source

Thrown at endpoint_storage/src/app.rs:192

    use tokio::io::AsyncReadExt;
    info!(%path, "downloading");
    let download_opts = DownloadOpts {
        kind: remote_storage::DownloadKind::Small,
        ..Default::default()
    };
    let mut body_read_buf = Vec::new();
    let stream = client
        .download(&path, &download_opts, &cancel)
        .await?
        .download_stream;
    tokio_util::io::StreamReader::new(stream)
        .read_to_end(&mut body_read_buf)
        .await?;
    let body_read = String::from_utf8(body_read_buf)?;
    if body != body_read {
        error!(%body, %body_read, "File contents do not match");
        anyhow::bail!("Read back file doesn't match original")
    }

    info!(%path, "removing");
    client.delete(&path, &cancel).await
}

fn bytes_to_stream(bytes: Bytes) -> impl futures::Stream<Item = std::io::Result<Bytes>> {
    futures::stream::once(futures::future::ready(Ok(bytes)))
}

#[cfg(test)]
mod tests {
    use super::*;
    use axum::{body::Body, extract::Request, response::Response};
    use http_body_util::BodyExt;
    use itertools::iproduct;
    use std::env::var;
    use std::sync::Arc;

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Give this instance its own bucket (or disable bucket sharing between deployments)
2. Remove or bypass caching proxies between the service and S3 so reads of a just-written key return the new object
3. Verify the storage_kind config (endpoint, bucket, credentials) matches the intended backend
4. As a last resort for k8s helm testing without S3, start with --no-s3-check-on-startup (understand that upload/download correctness is then unverified)

Example fix

# before: two deployments sharing bucket via caching proxy
storage_kind: { S3: { bucket_name: shared-bucket, ... } }
# after: dedicated bucket per deployment
storage_kind: { S3: { bucket_name: endpoint-storage-prod, ... } }

When it happens

Trigger: Starting endpoint_storage where the configured bucket/proxy returns stale or wrong content for the freshly-written probe object: a shared bucket behind an eventually-consistent or caching proxy layer, or misconfigured credentials pointing two services at the same keyspace.

Common situations: Multiple endpoint_storage instances proxying the same bucket (the very case the nanosecond-unique path tries to mitigate); an S3-compatible proxy (varnish/CDN/nginx) serving cached reads; pointing the check at a bucket with different credentials than the writer; local minio setups with proxying in front.

Related errors

• File size {request_len} exceeds max {max_len}
• uploading cancelled
• deleting cancelled
• deleting prefix cancelled
• Supply either config file path or --config=inline-config

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/a3c6e941ea4a9fed. Report an issue: GitHub.