apache/druid · warning

Resetting server[ ] with state[ ] as it is not syncing…

Error message

Resetting server[%s] with state[%s] as it is not syncing properly.

What it means

HttpServerInventoryView tracks Druid servers and their segment inventories via per-server syncers. When a server's ServerInventorySyncer reports it is not syncing properly (needsReset()), the view removes and re-adds the server with an empty segment list so the sync can restart cleanly. This log line warns that such a reset is being performed, which temporarily empties the server's inventory.

Solutions

1. Check logs on the affected data server for sync endpoint failures (full/delta sync errors) and fix the root cause (GC pauses, network, overloading).
2. Increase the sync poll interval or adjust unhealthy-hold/reset configuration so healthy-but-slow servers are not reset.
3. Verify the affected server is not overloaded; scale it or reduce segment counts to keep sync responses fast.
4. If resets are transient and self-heal, treat as noise; if persistent, restart the data server and investigate connectivity.

if (serverHolder.syncer.needsReset()) { log.warn("Server {} sync unhealthy; reset imminent", serverHolder.druidServer.getName()); }

When it happens

Trigger: The checkAndResetUnhealthyServers periodic task finds serverHolder.syncer.needsReset() true, typically because the server missed consecutive sync polls or returned malformed/failed sync responses (full or delta).

Common situations: Historical/broker processes overloaded and missing sync intervals; network partitions between coordinator-adjacent inventory consumers and data servers; data server restarts mid-sync; misconfigured sync polling intervals too aggressive for cluster size.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Not adding or running callbacks for existing segment
• Not running cleanup or callbacks for non-existing segment
• A-Not-B requires at least 1 sketch
• Access-Check-Result
• Action [ ] failed for worker [ ] with status ( )

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/9334912720c22860. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/client/HttpServerInventoryView.java:483

          e.getKey(),
          serverHolder.syncer.getDebugInfo()
      );
    }
    return result;
  }

  @VisibleForTesting
  void checkAndResetUnhealthyServers()
  {
    // Ensure that the collection is not being modified during iteration. Iterate over a copy
    final Set<Map.Entry<String, DruidServerHolder>> serverEntrySet = ImmutableSet.copyOf(servers.entrySet());
    for (Map.Entry<String, DruidServerHolder> e : serverEntrySet) {
      DruidServerHolder serverHolder = e.getValue();
      if (serverHolder.syncer.needsReset()) {
        synchronized (servers) {
          // Reset only if the server is still present in the map
          if (servers.containsKey(e.getKey())) {
            log.warn(
                "Resetting server[%s] with state[%s] as it is not syncing properly.",
                serverHolder.druidServer.getName(),
                serverHolder.syncer.getDebugInfo()
            );
            serverRemoved(serverHolder.druidServer);
            serverAdded(serverHolder.druidServer.copyWithoutSegments());
          }
        }
      }
    }
  }

  private void emitServerStatusMetrics()
  {
    final ServiceMetricEvent.Builder eventBuilder = ServiceMetricEvent.builder();
    try {
      final Map<String, DruidServerHolder> serversCopy = ImmutableMap.copyOf(servers);
      serversCopy.forEach((serverName, serverHolder) -> {

View on GitHub (pinned to 9b90983fd2)