nautechsystems/nautilus_trader · error · anyhow::Error

recovery reconcile failed: both REST queries returned errors

Error message

recovery reconcile failed: both REST queries returned errors

What it means

During listen-key recovery the adapter reconciles state by querying open orders and open algo orders over REST and emitting OrderStatusReports, so nothing is missed across the stream gap. If both the open-orders and open-algo-orders queries fail, the adapter cannot confirm account state and bails, causing the whole recovery (including the freshly created listen key) to be retried with backoff.

Solutions

  1. Check the log lines just above the bail for each query's specific error and fix that root cause (clock sync via NTP, API key permissions, network/proxy)
  2. Verify system time is within Binance's recvWindow tolerance (sync with NTP; timestamp errors fail signed requests)
  3. Check Binance status page / rate-limit headers for outages or bans; the recovery loop retries with backoff so transient outages self-heal
  4. Confirm the API key has Futures read access and the correct base URL (production vs testnet) for the environment

Example fix

// before: client with local clock drift causing -1021 on both queries
let client = BinanceFuturesHttpClient::new(key, secret, None);
// after: keep host clock synchronized (NTP) or configure an explicit time offset
// sudo timedatectl set-ntp true
let client = BinanceFuturesHttpClient::new(key, secret, Some(time_offset_ms));
Defensive patterns

Strategy: try-catch

Validate before calling

// Before starting the node, verify signed REST access works and clock is in sync:
import time, hmac, hashlib, requests
server = requests.get('https://fapi.binance.com/fapi/v1/time').json()['serverTime']
drift = abs(int(time.time()*1000) - server)
assert drift < 1000, f'clock drift {drift}ms exceeds Binance tolerance'

Try / catch

// The internal recovery loop retries with backoff; wrap your node startup in a
// connectivity pre-check and surface alerts on repeated reconcile failures:
try { await client.query_open_orders(); await client.query_open_algo_orders(); }
catch (e) { log.error("REST reconcile pre-check failed, fix before trading: " + e); }

Prevention

When it happens

Trigger: emit_open_order_reports runs during recover_user_data_stream and both REST calls (query open orders, query open algo orders) return errors — e.g. network outage, Binance API -1021 timestamp errors, invalid API key/permissions, or rate-limit bans.

Common situations: Binance API downtime or regional blocks; system clock drift causing signed-request timestamp rejection; API key lacking futures read permissions; hitting request-weight rate limits during volatile markets.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors


AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/0f3974682e9f570d. Report an issue: GitHub.

Appendix: source

Thrown at crates/adapters/binance/src/futures/websocket/streams/recovery.rs:427

                    }
                    Err(e) => {
                        log::warn!(
                            "Failed to build OrderStatusReport for algo {} during recovery reconcile: {e}",
                            algo_order.symbol,
                        );
                    }
                }
            }
            true
        }
        Err(e) => {
            log::warn!("Failed to query open algo orders for recovery reconcile: {e}");
            false
        }
    };

    if !open_ok && !algo_ok {
        anyhow::bail!("recovery reconcile failed: both REST queries returned errors");
    }

    log::info!("Recovery reconcile emitted {emitted} OrderStatusReport(s)");
    Ok(())
}

fn resolve_precision(
    instruments: &DashMap<ustr::Ustr, crate::futures::http::client::BinanceFuturesInstrument>,
    symbol_ustr: &ustr::Ustr,
    product_type: BinanceProductType,
) -> anyhow::Result<(InstrumentId, u8, u8)> {
    let instrument_id = format_instrument_id(symbol_ustr, product_type);
    let instrument = instruments
        .get(symbol_ustr)
        .map(|instrument| instrument.value().clone())
        .with_context(|| format!("missing instrument metadata for {instrument_id}"))?;
    anyhow::ensure!(
        instrument.id() == instrument_id,

View on GitHub (pinned to 18893faf8b)