{"record":{"id":"8528cba2284b3eec","repo":"netdata/netdata","slug":"remote-object-not-found-key","errorCode":null,"errorMessage":"remote object not found: {key}","messagePattern":"remote object not found: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/crates/otel-ledger/src/ledger/rpc/handler.rs","lineNumber":684,"sourceCode":"            }\n        }\n    }\n\n    serde_json::to_vec(&serde_json::Value::Object(map)).ok()\n}\n\n/// Convert a remote-read failure into the `anyhow::Error` handed to the\n/// file-cache — which logs it verbatim with `{e:#}`.\n///\n/// MUST flatten through `StorageError`'s `Display`, never extract the raw\n/// inner error: `Display` renders the full source chain with URL query\n/// strings redacted (`file-lifecycle`'s `redact`), and a raw chain would put\n/// request credentials in the journal (AWS carries the STS web-identity JWT\n/// and request signatures in URL queries). Nothing is lost by flattening —\n/// `Display` already carries every chain level as text.\nfn read_error_to_anyhow(key: &str, e: StorageError) -> anyhow::Error {\n    match e {\n        StorageError::NotFound => anyhow::anyhow!(\"remote object not found: {key}\"),\n        other => anyhow::anyhow!(\"remote read failed for {key}: {other}\"),\n    }\n}\n\n#[cfg(test)]\nmod tests;\n","sourceCodeStart":666,"sourceCodeEnd":691,"githubUrl":"https://github.com/netdata/netdata/blob/4864de85e26f6734d92cfc27ccbeff5921f49938/src/crates/otel-ledger/src/ledger/rpc/handler.rs#L666-L691","documentation":"The otel-ledger RPC handler maps a remote-read StorageError::NotFound into this anyhow error handed to the file-cache, which logs it with full context ('{e:#}'). It specifically names the missing object key so operators can tell 'the remote object is gone' apart from 'the remote read failed' (auth, network, etc.), which gets a different message. The comment is explicit that errors must be flattened through StorageError's Display so URL query strings (carrying STS tokens/signatures) are redacted before reaching the journal.","triggerScenarios":"A ledger remote read for a key that the backend reports as NotFound — e.g. an SFST object deleted by retention/lifecycle rules, a race where another node evicted it, or a catalog entry pointing at an object never uploaded after a crashed seal.","commonSituations":"Remote lifecycle policies (S3 expiration) deleting objects the catalog still references; partial uploads after crashes; multi-node setups racing on eviction; manually deleted buckets/prefixes.","solutions":["Verify the object exists in the remote backend for the named key (with redacted/credential-safe tooling)","If a lifecycle rule is deleting objects, align it with the ledger's retention so the catalog and storage agree","Rebuild or repair the catalog entry if the object was lost and the data is re-derivable","Never widen the error to include raw request URLs — the redaction through StorageError's Display is load-bearing (URLs carry STS JWTs/signatures)"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"# before reprocessing, confirm the object still exists (credentials stay in env, never in logs)\naws s3api head-object --bucket \"$BUCKET\" --key \"$KEY\" >/dev/null && echo exists || echo missing","typeGuard":null,"tryCatchPattern":"treat 'remote object not found: {key}' as a skip-and-continue condition: log the key, drop or repair the catalog entry, and keep processing; distinguish it from 'remote read failed for {key}' which signals transient/auth failure worth retrying","preventionTips":["Align remote lifecycle/expiration rules with ledger retention so storage and catalog agree","Never log raw error chains from the storage layer — the StorageError Display redaction is what keeps STS tokens out of the journal"],"tags":["storage","not-found","ledger","otel","remote-storage","object-store"],"backgroundTag":null,"analyzedSha":"4864de85e26f6734d92cfc27ccbeff5921f49938","analyzedAt":"2026-08-15T09:12:38.226Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}