{"record":{"id":"0f3974682e9f570d","repo":"nautechsystems/nautilus_trader","slug":"recovery-reconcile-failed-both-rest-queries-retur","errorCode":null,"errorMessage":"recovery reconcile failed: both REST queries returned errors","messagePattern":"recovery reconcile failed: both REST queries returned errors","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/binance/src/futures/websocket/streams/recovery.rs","lineNumber":427,"sourceCode":"                    }\n                    Err(e) => {\n                        log::warn!(\n                            \"Failed to build OrderStatusReport for algo {} during recovery reconcile: {e}\",\n                            algo_order.symbol,\n                        );\n                    }\n                }\n            }\n            true\n        }\n        Err(e) => {\n            log::warn!(\"Failed to query open algo orders for recovery reconcile: {e}\");\n            false\n        }\n    };\n\n    if !open_ok && !algo_ok {\n        anyhow::bail!(\"recovery reconcile failed: both REST queries returned errors\");\n    }\n\n    log::info!(\"Recovery reconcile emitted {emitted} OrderStatusReport(s)\");\n    Ok(())\n}\n\nfn resolve_precision(\n    instruments: &DashMap<ustr::Ustr, crate::futures::http::client::BinanceFuturesInstrument>,\n    symbol_ustr: &ustr::Ustr,\n    product_type: BinanceProductType,\n) -> anyhow::Result<(InstrumentId, u8, u8)> {\n    let instrument_id = format_instrument_id(symbol_ustr, product_type);\n    let instrument = instruments\n        .get(symbol_ustr)\n        .map(|instrument| instrument.value().clone())\n        .with_context(|| format!(\"missing instrument metadata for {instrument_id}\"))?;\n    anyhow::ensure!(\n        instrument.id() == instrument_id,","sourceCodeStart":409,"sourceCodeEnd":445,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/binance/src/futures/websocket/streams/recovery.rs#L409-L445","documentation":"During listen-key recovery the adapter reconciles state by querying open orders and open algo orders over REST and emitting OrderStatusReports, so nothing is missed across the stream gap. If both the open-orders and open-algo-orders queries fail, the adapter cannot confirm account state and bails, causing the whole recovery (including the freshly created listen key) to be retried with backoff.","triggerScenarios":"emit_open_order_reports runs during recover_user_data_stream and both REST calls (query open orders, query open algo orders) return errors — e.g. network outage, Binance API -1021 timestamp errors, invalid API key/permissions, or rate-limit bans.","commonSituations":"Binance API downtime or regional blocks; system clock drift causing signed-request timestamp rejection; API key lacking futures read permissions; hitting request-weight rate limits during volatile markets.","solutions":["Check the log lines just above the bail for each query's specific error and fix that root cause (clock sync via NTP, API key permissions, network/proxy)","Verify system time is within Binance's recvWindow tolerance (sync with NTP; timestamp errors fail signed requests)","Check Binance status page / rate-limit headers for outages or bans; the recovery loop retries with backoff so transient outages self-heal","Confirm the API key has Futures read access and the correct base URL (production vs testnet) for the environment"],"exampleFix":"// before: client with local clock drift causing -1021 on both queries\nlet client = BinanceFuturesHttpClient::new(key, secret, None);\n// after: keep host clock synchronized (NTP) or configure an explicit time offset\n// sudo timedatectl set-ntp true\nlet client = BinanceFuturesHttpClient::new(key, secret, Some(time_offset_ms));","handlingStrategy":"try-catch","validationCode":"// Before starting the node, verify signed REST access works and clock is in sync:\nimport time, hmac, hashlib, requests\nserver = requests.get('https://fapi.binance.com/fapi/v1/time').json()['serverTime']\ndrift = abs(int(time.time()*1000) - server)\nassert drift < 1000, f'clock drift {drift}ms exceeds Binance tolerance'","typeGuard":null,"tryCatchPattern":"// The internal recovery loop retries with backoff; wrap your node startup in a\n// connectivity pre-check and surface alerts on repeated reconcile failures:\ntry { await client.query_open_orders(); await client.query_open_algo_orders(); }\ncatch (e) { log.error(\"REST reconcile pre-check failed, fix before trading: \" + e); }","preventionTips":["Run NTP time sync — timestamp drift is the most common cause of signed-request failures","Grant the API key Futures read permissions and use the correct base URL (prod vs testnet)","Watch Binance status/rate-limit headers; back off during weight-limit bans","Check connectivity/proxy to fapi.binance.com before live sessions"],"tags":["rest-api","recovery","reconcile","network","binance-futures"],"backgroundTag":"api-request-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}