What causes "recovery reconcile failed: both REST queries returned errors" in nautechsystems/nautilus_trader?

Read the two preceding warn logs ('Failed to query open orders...' / 'Failed to query open algo orders...') which carry the actual REST errors

nautechsystems/nautilus_trader · error · anyhow::Error

recovery reconcile failed: both REST queries returned errors

Error message

recovery reconcile failed: both REST queries returned errors

What it means

After a private-stream reconnect or listenKey rotation, recovery reconciles open state with two REST queries: query_open_orders (regular) and query_open_algo_orders (conditional). Each individual failure is only logged as a warning; recovery aborts with this error only when BOTH fail, and recover_with_retry schedules another attempt, so open-order state may lag until at least one query succeeds.

Source

Thrown at crates/adapters/binance/src/futures/websocket/streams/recovery.rs:347

                    }
                    Err(e) => {
                        log::warn!(
                            "Failed to build OrderStatusReport for algo {} during recovery reconcile: {e}",
                            algo_order.symbol,
                        );
                    }
                }
            }
            true
        }
        Err(e) => {
            log::warn!("Failed to query open algo orders for recovery reconcile: {e}");
            false
        }
    };

    if !open_ok && !algo_ok {
        anyhow::bail!("recovery reconcile failed: both REST queries returned errors");
    }

    log::info!("Recovery reconcile emitted {emitted} OrderStatusReport(s)");
    Ok(())
}

fn resolve_precision(
    instruments: &dashmap::DashMap<
        ustr::Ustr,
        crate::futures::http::client::BinanceFuturesInstrument,
    >,
    symbol_ustr: &ustr::Ustr,
    product_type: BinanceProductType,
) -> (InstrumentId, u8, u8) {
    if let Some(instrument) = instruments.get(symbol_ustr) {
        (
            instrument.id(),
            instrument.price_precision() as u8,

View on GitHub (pinned to a4b06ed870)

Solutions

Read the two preceding warn logs ('Failed to query open orders...' / 'Failed to query open algo orders...') which carry the actual REST errors
Fix whatever they show: credentials/permissions (Futures enabled, IP whitelist), network/proxy, or system clock drift affecting signed requests
If rate-limited (429/418), reduce request pressure and wait out the ban window; recovery retries automatically
If persistent, manually verify GET /fapi/v1/openOrders works from this host with the same key, then restart the node

Defensive patterns

Strategy: retry

Try / catch

Rely on the built-in retry-on-failure recovery; capture the two warn-level per-query errors that precede it to classify causes (auth vs rate limit vs network) and alert if 'both REST queries returned errors' persists beyond a few attempts.

Prevention

Keep API key permissions and IP whitelist valid for Futures REST
Watch rate-limit headers and reduce polling pressure to avoid 429/418 during rotations
Alert on repeated reconcile failures so open-order state gaps do not go unnoticed

When it happens

Trigger: Both query_open_orders(params) and query_open_algo_orders(None) fail on the same recovery attempt: network outage, 401 (invalid key / no Futures permission / IP ban), 429 or 418 rate limiting, or Binance 5xx errors.

Common situations: Connectivity blip during listenKey rotation; API key permissions changed while the node was running; request pressure triggering rate limits; a Binance-side incident taking both REST endpoints down.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@a4b06ed870 (2026-08-16). Data as JSON: /api/errors/0f3974682e9f570d. Report an issue: GitHub.