{"record":{"id":"e8e69dfbf8204d5f","repo":"neondatabase/neon","slug":"missing-etag-header","errorCode":null,"errorMessage":"Missing ETag header","messagePattern":"Missing ETag header","errorType":"exception","errorClass":"DownloadError","httpStatus":null,"severity":"error","filePath":"libs/remote_storage/src/gcs_bucket.rs","lineNumber":843,"sourceCode":"                crate::metrics::BUCKET_METRICS.req_seconds.observe_elapsed(\n                    kind,\n                    AttemptOutcome::Err,\n                    started_at,\n                );\n\n                return Err(DownloadError::Other(\n                    anyhow::Error::new(e).context(\"download s3 object\"),\n                ));\n            }\n        };\n\n        let remaining = self.timeout.saturating_sub(started_at.elapsed());\n\n        let metadata = resp.metadata.map(StorageMetadata);\n\n        let etag = resp\n            .etag\n            .ok_or(DownloadError::Other(anyhow::anyhow!(\"Missing ETag header\")))?\n            .into();\n\n        let last_modified: SystemTime = to_system_time(resp.updated).unwrap_or(SystemTime::now());\n\n        // But let data stream pass through\n        Ok(Download {\n            download_stream: Box::pin(object_output.bytes_stream().map(|item| {\n                item.map_err(|e: reqwest::Error| std::io::Error::new(std::io::ErrorKind::Other, e))\n            })),\n            etag,\n            last_modified,\n            metadata,\n        })\n    }\n    \n    async fn copy_object(\n        &self, \n        from: &RemotePath,","sourceCodeStart":825,"sourceCodeEnd":861,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/remote_storage/src/gcs_bucket.rs#L825-L861","documentation":"After a successful media GET, the backend builds the Download from the earlier-parsed object metadata and requires an ETag; if the etag field was absent from the metadata JSON this error fires. GCS always returns an ETag for objects, so despite the 'header' wording this signals an unexpected response shape (a proxy/gateway omitting the field, or an API change) rather than a normal GCS response missing a header.","triggerScenarios":"download() where the parsed GCSObject metadata has etag == None — a GCS-compatible gateway or emulator dropped the field, an API shape change, or an error body that happened to parse as an object.","commonSituations":"S3/GCS-compatible proxies or emulators in front of storage that omit etag in JSON responses; rare API anomalies; migrations between endpoint styles.","solutions":["Capture the raw metadata JSON body and confirm whether etag is genuinely absent versus a deserialization mismatch","Compare the response from real storage.googleapis.com against any intermediary in the path","Retry once — if an anomalous body was transient, the retry succeeds","If a gateway legitimately omits etag, fix its field mapping or make the backend tolerate absence (Option<ETag>)"],"exampleFix":"// before: absent etag aborts the whole download\nlet etag = resp.etag.ok_or(DownloadError::Other(anyhow::anyhow!(\"Missing ETag header\")))?.into();\n\n// after: tolerate absence with a generated fallback identifier\nlet etag = match resp.etag {\n    Some(e) => ETag::from(e),\n    None => ETag::from(format!(\"\\\"gen-{}\\\"\", generation)),\n};","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Unexpected response shape: surface with the key, don't loop retrying.\nmatch storage.download(from, &cancel).await {\n    Ok(dl) => Ok(dl),\n    Err(DownloadError::Other(e)) if format!(\"{e:#}\").contains(\"Missing ETag header\") => {\n        Err(anyhow::anyhow!(\"metadata for {from} lacked etag; check gateways/proxies in the GCS path: {e:#}\"))\n    }\n    Err(e) => Err(e.into()),\n}","preventionTips":["Do not put field-dropping proxies between the client and GCS","Monitor for this error after any endpoint/gateway change — it detects silent response-shape drift","Keep an integration test that downloads a real object end-to-end through your production path"],"tags":["gcs","google-cloud-storage","etag","metadata","deserialization"],"backgroundTag":"missing-response-field","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}