{"record":{"id":"a3c6e941ea4a9fed","repo":"neondatabase/neon","slug":"read-back-file-doesn-t-match-original","errorCode":null,"errorMessage":"Read back file doesn't match original","messagePattern":"Read back file doesn't match original","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"endpoint_storage/src/app.rs","lineNumber":192,"sourceCode":"\n    use tokio::io::AsyncReadExt;\n    info!(%path, \"downloading\");\n    let download_opts = DownloadOpts {\n        kind: remote_storage::DownloadKind::Small,\n        ..Default::default()\n    };\n    let mut body_read_buf = Vec::new();\n    let stream = client\n        .download(&path, &download_opts, &cancel)\n        .await?\n        .download_stream;\n    tokio_util::io::StreamReader::new(stream)\n        .read_to_end(&mut body_read_buf)\n        .await?;\n    let body_read = String::from_utf8(body_read_buf)?;\n    if body != body_read {\n        error!(%body, %body_read, \"File contents do not match\");\n        anyhow::bail!(\"Read back file doesn't match original\")\n    }\n\n    info!(%path, \"removing\");\n    client.delete(&path, &cancel).await\n}\n\nfn bytes_to_stream(bytes: Bytes) -> impl futures::Stream<Item = std::io::Result<Bytes>> {\n    futures::stream::once(futures::future::ready(Ok(bytes)))\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use axum::{body::Body, extract::Request, response::Response};\n    use http_body_util::BodyExt;\n    use itertools::iproduct;\n    use std::env::var;\n    use std::sync::Arc;","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/endpoint_storage/src/app.rs#L174-L210","documentation":"check_storage_permissions (run at endpoint_storage startup unless --no-s3-check-on-startup) writes a uniquely-named probe file to the remote bucket, downloads it back, and compares the strings. A mismatch means the storage backend is not faithfully returning what was written — for example multiple instances sharing one bucket through a caching proxy — and the check bails after logging both bodies.","triggerScenarios":"Starting endpoint_storage where the configured bucket/proxy returns stale or wrong content for the freshly-written probe object: a shared bucket behind an eventually-consistent or caching proxy layer, or misconfigured credentials pointing two services at the same keyspace.","commonSituations":"Multiple endpoint_storage instances proxying the same bucket (the very case the nanosecond-unique path tries to mitigate); an S3-compatible proxy (varnish/CDN/nginx) serving cached reads; pointing the check at a bucket with different credentials than the writer; local minio setups with proxying in front.","solutions":["Give this instance its own bucket (or disable bucket sharing between deployments)","Remove or bypass caching proxies between the service and S3 so reads of a just-written key return the new object","Verify the storage_kind config (endpoint, bucket, credentials) matches the intended backend","As a last resort for k8s helm testing without S3, start with --no-s3-check-on-startup (understand that upload/download correctness is then unverified)"],"exampleFix":"# before: two deployments sharing bucket via caching proxy\nstorage_kind: { S3: { bucket_name: shared-bucket, ... } }\n# after: dedicated bucket per deployment\nstorage_kind: { S3: { bucket_name: endpoint-storage-prod, ... } }","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run one endpoint_storage deployment per bucket; never share buckets between proxies","Keep caching layers out of the read path for freshly written keys, or bust cache on the probe key","Alert on startup check failure: it signals silent data corruption ahead, not a transient blip"],"tags":["endpoint-storage","s3","startup-check","data-integrity","rust"],"backgroundTag":"data-integrity-check-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}