nautechsystems/nautilus_trader · error · anyhow::Error
recovery reconcile failed: both REST queries returned errors
Error message
recovery reconcile failed: both REST queries returned errors
What it means
During listen-key recovery the adapter reconciles state by querying open orders and open algo orders over REST and emitting OrderStatusReports, so nothing is missed across the stream gap. If both the open-orders and open-algo-orders queries fail, the adapter cannot confirm account state and bails, causing the whole recovery (including the freshly created listen key) to be retried with backoff.
Solutions
- Check the log lines just above the bail for each query's specific error and fix that root cause (clock sync via NTP, API key permissions, network/proxy)
- Verify system time is within Binance's recvWindow tolerance (sync with NTP; timestamp errors fail signed requests)
- Check Binance status page / rate-limit headers for outages or bans; the recovery loop retries with backoff so transient outages self-heal
- Confirm the API key has Futures read access and the correct base URL (production vs testnet) for the environment
Example fix
// before: client with local clock drift causing -1021 on both queries let client = BinanceFuturesHttpClient::new(key, secret, None); // after: keep host clock synchronized (NTP) or configure an explicit time offset // sudo timedatectl set-ntp true let client = BinanceFuturesHttpClient::new(key, secret, Some(time_offset_ms));
Defensive patterns
Strategy: try-catch
Validate before calling
// Before starting the node, verify signed REST access works and clock is in sync:
import time, hmac, hashlib, requests
server = requests.get('https://fapi.binance.com/fapi/v1/time').json()['serverTime']
drift = abs(int(time.time()*1000) - server)
assert drift < 1000, f'clock drift {drift}ms exceeds Binance tolerance' Try / catch
// The internal recovery loop retries with backoff; wrap your node startup in a
// connectivity pre-check and surface alerts on repeated reconcile failures:
try { await client.query_open_orders(); await client.query_open_algo_orders(); }
catch (e) { log.error("REST reconcile pre-check failed, fix before trading: " + e); } Prevention
- Run NTP time sync — timestamp drift is the most common cause of signed-request failures
- Grant the API key Futures read permissions and use the correct base URL (prod vs testnet)
- Watch Binance status/rate-limit headers; back off during weight-limit bans
- Check connectivity/proxy to fapi.binance.com before live sessions
When it happens
Trigger: emit_open_order_reports runs during recover_user_data_stream and both REST calls (query open orders, query open algo orders) return errors — e.g. network outage, Binance API -1021 timestamp errors, invalid API key/permissions, or rate-limit bans.
Common situations: Binance API downtime or regional blocks; system clock drift causing signed-request timestamp rejection; API key lacking futures read permissions; hitting request-weight rate limits during volatile markets.
Understand the failure class
Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.
Related errors
- failed to connect Binance Futures private WebSocket
- {e}
- failed to connect Binance Futures market WebSocket
- failed to connect Binance Futures public WebSocket
- Failed to fetch products
AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08).
Data as JSON: /api/errors/0f3974682e9f570d.
Report an issue: GitHub.
Appendix: source
Thrown at crates/adapters/binance/src/futures/websocket/streams/recovery.rs:427
}
Err(e) => {
log::warn!(
"Failed to build OrderStatusReport for algo {} during recovery reconcile: {e}",
algo_order.symbol,
);
}
}
}
true
}
Err(e) => {
log::warn!("Failed to query open algo orders for recovery reconcile: {e}");
false
}
};
if !open_ok && !algo_ok {
anyhow::bail!("recovery reconcile failed: both REST queries returned errors");
}
log::info!("Recovery reconcile emitted {emitted} OrderStatusReport(s)");
Ok(())
}
fn resolve_precision(
instruments: &DashMap<ustr::Ustr, crate::futures::http::client::BinanceFuturesInstrument>,
symbol_ustr: &ustr::Ustr,
product_type: BinanceProductType,
) -> anyhow::Result<(InstrumentId, u8, u8)> {
let instrument_id = format_instrument_id(symbol_ustr, product_type);
let instrument = instruments
.get(symbol_ustr)
.map(|instrument| instrument.value().clone())
.with_context(|| format!("missing instrument metadata for {instrument_id}"))?;
anyhow::ensure!(
instrument.id() == instrument_id,View on GitHub (pinned to 18893faf8b)