{"record":{"id":"cecc6fd6c5a1a78e","repo":"zeroclaw-labs/zeroclaw","slug":"node-request-failed","errorCode":null,"errorMessage":"Node request failed: {} {}","messagePattern":"Node request failed: (.+?) (.+?)","errorType":"http","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/zeroclaw-runtime/src/nodes/transport.rs","lineNumber":108,"sourceCode":"        let body = serde_json::to_vec(&payload)?;\n        let timestamp = Utc::now().timestamp();\n        let nonce = uuid::Uuid::new_v4().to_string();\n        let signature = sign_request(&self.shared_secret, &body, timestamp, &nonce)?;\n\n        let url = format!(\"https://{node_address}/api/node-control/{endpoint}\");\n        let resp = self\n            .http\n            .post(&url)\n            .header(\"X-ZeroClaw-Timestamp\", timestamp.to_string())\n            .header(\"X-ZeroClaw-Nonce\", &nonce)\n            .header(\"X-ZeroClaw-Signature\", &signature)\n            .header(\"Content-Type\", \"application/json\")\n            .body(body)\n            .send()\n            .await?;\n\n        if !resp.status().is_success() {\n            bail!(\n                \"Node request failed: {} {}\",\n                resp.status(),\n                resp.text().await.unwrap_or_default()\n            );\n        }\n\n        Ok(resp.json().await?)\n    }\n\n    /// Verify an incoming request from a peer node.\n    pub fn verify_incoming(\n        &self,\n        payload: &[u8],\n        timestamp_header: &str,\n        nonce_header: &str,\n        signature_header: &str,\n    ) -> Result<bool> {\n        let timestamp: i64 = timestamp_header.parse().map_err(|_| {","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-runtime/src/nodes/transport.rs#L90-L126","documentation":"NodeTransport::send signs a JSON payload and POSTs it to https://{node_address}/api/node-control/{endpoint}. Network-level failures (DNS, connect, TLS) propagate as reqwest errors through `?`; this bail fires only after a response arrives with a non-2xx status, and embeds both the HTTP status and the peer's response body. It tells you the peer answered and refused.","triggerScenarios":"401/403: signature/secret verification rejected on the peer (shared secrets differ) or the timestamp window (error 867) fired; 404: wrong node_address or unknown endpoint segment; 500/502/503: peer-side handler crashed, upstream gateway error, peer under maintenance; the peer returned an HTML error page, which appears verbatim in the message.","commonSituations":"Nodes provisioned with different shared secrets (rotated on one side only); reverse proxy in front of the node rewriting paths; peer process half-dead so its gateway returns 502; typos in the endpoint string.","solutions":["Read the embedded status: 401/403 → compare and re-distribute the shared secret on both nodes; 404 → verify node_address and endpoint name; 5xx → inspect the peer node's logs","Confirm both nodes run compatible versions exposing /api/node-control/{endpoint}","If a proxy fronts the peer, check it passes the X-ZeroClaw-* headers and path unchanged","Retry transient 5xx with backoff; do not retry 4xx — those are deterministic rejections"],"exampleFix":"// before: fire and forget\nlet v = transport.send(&addr, \"exec\", payload).await?;\n\n// after: classify the failure\nmatch transport.send(&addr, \"exec\", payload).await {\n    Ok(v) => v,\n    Err(e) if e.to_string().contains(\"Node request failed: 40\") => {\n        log::warn!(\"auth/path problem: {e}\");\n        return Err(e);\n    }\n    Err(e) => { retry_with_backoff(...).await? }\n}","handlingStrategy":"try-catch","validationCode":"// optional preflight: cheap health probe before the signed call\nlet resp = reqwest::get(format!(\"https://{node_address}/api/node-control/health\")).await;\nif !resp.map(|r| r.status().is_success()).unwrap_or(false) {\n    // defer the signed request; peer is unhealthy or unreachable\n}","typeGuard":null,"tryCatchPattern":"match transport.send(node_address, endpoint, payload).await {\n    Ok(v) => v,\n    Err(e) => {\n        let msg = e.to_string();\n        if msg.contains(\"Node request failed: 401\") || msg.contains(\": 403\") {\n            // deterministic auth failure: fix/rotate shared secret; do not retry\n        } else if msg.contains(\"Node request failed: 5\") {\n            // transient: retry with bounded exponential backoff\n        } else {\n            return Err(e); // network-level reqwest error or unknown status\n        }\n    }\n}","preventionTips":["Distribute the shared secret via one mechanism and rotate it on both sides atomically","Health-check peer nodes before batches of signed requests","Log status-code histograms to distinguish auth problems from peer outages","Never retry 4xx responses — they are deterministic"],"tags":["rust","zeroclaw","http","nodes","distributed","api-client"],"backgroundTag":"http-request-failed","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}