netdata/netdata · warning

remote object not found: {key}

Error message

remote object not found: {key}

What it means

The otel-ledger RPC handler maps a remote-read StorageError::NotFound into this anyhow error handed to the file-cache, which logs it with full context ('{e:#}'). It specifically names the missing object key so operators can tell 'the remote object is gone' apart from 'the remote read failed' (auth, network, etc.), which gets a different message. The comment is explicit that errors must be flattened through StorageError's Display so URL query strings (carrying STS tokens/signatures) are redacted before reaching the journal.

Source

Thrown at src/crates/otel-ledger/src/ledger/rpc/handler.rs:684

            }
        }
    }

    serde_json::to_vec(&serde_json::Value::Object(map)).ok()
}

/// Convert a remote-read failure into the `anyhow::Error` handed to the
/// file-cache — which logs it verbatim with `{e:#}`.
///
/// MUST flatten through `StorageError`'s `Display`, never extract the raw
/// inner error: `Display` renders the full source chain with URL query
/// strings redacted (`file-lifecycle`'s `redact`), and a raw chain would put
/// request credentials in the journal (AWS carries the STS web-identity JWT
/// and request signatures in URL queries). Nothing is lost by flattening —
/// `Display` already carries every chain level as text.
fn read_error_to_anyhow(key: &str, e: StorageError) -> anyhow::Error {
    match e {
        StorageError::NotFound => anyhow::anyhow!("remote object not found: {key}"),
        other => anyhow::anyhow!("remote read failed for {key}: {other}"),
    }
}

#[cfg(test)]
mod tests;

View on GitHub (pinned to 4864de85e2)

Solutions

1. Verify the object exists in the remote backend for the named key (with redacted/credential-safe tooling)
2. If a lifecycle rule is deleting objects, align it with the ledger's retention so the catalog and storage agree
3. Rebuild or repair the catalog entry if the object was lost and the data is re-derivable
4. Never widen the error to include raw request URLs — the redaction through StorageError's Display is load-bearing (URLs carry STS JWTs/signatures)

# before reprocessing, confirm the object still exists (credentials stay in env, never in logs)
aws s3api head-object --bucket "$BUCKET" --key "$KEY" >/dev/null && echo exists || echo missing

treat 'remote object not found: {key}' as a skip-and-continue condition: log the key, drop or repair the catalog entry, and keep processing; distinguish it from 'remote read failed for {key}' which signals transient/auth failure worth retrying

When it happens

Trigger: A ledger remote read for a key that the backend reports as NotFound — e.g. an SFST object deleted by retention/lifecycle rules, a race where another node evicted it, or a catalog entry pointing at an object never uploaded after a crashed seal.

Common situations: Remote lifecycle policies (S3 expiration) deleting objects the catalog still references; partial uploads after crashes; multi-node setups racing on eviction; manually deleted buckets/prefixes.

Related errors

• base_dir must be set (the mandatory root for all signal stor
• remote_storage.uri must be set when remote_storage.enabled i
• logs.{e}
• traces.{e}
• WAL payload format {found} is not the traces proof format {T

AI-assisted analysis of netdata/netdata@4864de85e2 (2026-08-15). Data as JSON: /api/errors/8528cba2284b3eec. Report an issue: GitHub.