phacility/phabricator · error · Exception

Leader lost: no up-to-date nodes in repository cluster are f

Error message

Leader lost: no up-to-date nodes in repository cluster are fetchable.

What it means

This device is behind and needs to fetch from an up-to-date node. Version rows say up-to-date devices exist, but after filtering (bindings whose device is not in the current device set, and bindings whose protocol is not 'ssh', since the TODO notes only SSH fetching is supported) nothing is fetchable. The engine treats the leader as lost and throws rather than fetch stale data.

Source

Thrown at src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php:715

    $fetchable = array();
    foreach ($bindings as $binding) {
      // We can't fetch from nodes which don't have the newest version.
      $device_phid = $binding->getDevicePHID();
      if (empty($device_map[$device_phid])) {
        continue;
      }

      // TODO: For now, only fetch over SSH. We could support fetching over
      // HTTP eventually.
      if ($binding->getAlmanacPropertyValue('protocol') != 'ssh') {
        continue;
      }

      $fetchable[] = $binding;
    }

    if (!$fetchable) {
      throw new Exception(
        pht(
          'Leader lost: no up-to-date nodes in repository cluster are '.
          'fetchable.'));
    }

    // If we can synchronize from multiple sources, choose one at random.
    shuffle($fetchable);

    $caught = null;
    foreach ($fetchable as $binding) {
      try {
        $this->synchronizeWorkingCopyFromBinding(
          $binding,
          $local_version,
          $remote_version);
        $caught = null;
        break;
      } catch (Exception $ex) {

View on GitHub (pinned to 5720a38cfe)

Solutions

1. Give at least one up-to-date device an active binding with protocol 'ssh' and confirm that host is reachable from the lagging device
2. Re-bind or bring back online the up-to-date device that holds the newest version
3. If no up-to-date copy is recoverable, this is data loss: follow the Cluster: Repositories documentation to rebuild from the best surviving copy and reset the version rows deliberately

// before demoting or removing nodes, assert an up-to-date ssh binding remains
$fetchable = array();
foreach ($service->getActiveBindings() as $binding) {
  if ($binding->getAlmanacPropertyValue('protocol') === 'ssh'
      && isset($up_to_date[$binding->getDevicePHID()])) {
    $fetchable[] = $binding;
  }
}
if (!$fetchable) {
  // refuse the maintenance action: it would strand lagging devices
}

function hasFetchableUpToDateBinding(array $bindings, array $up_to_date_phids) {
  foreach ($bindings as $binding) {
    if ($binding->getAlmanacPropertyValue('protocol') === 'ssh'
        && in_array($binding->getDevicePHID(), $up_to_date_phids, true)) {
      return true;
    }
  }
  return false;
}

try {
  $engine->synchronizeWorkingCopyFromDevices($phids, $local, $remote);
} catch (Exception $ex) {
  // data availability problem: alert operators, do not silently fall back to stale data
}

When it happens

Trigger: Every up-to-date device is bound with a protocol other than ssh (for example all bindings left at http), or the up-to-date devices' bindings are disabled/removed while their version rows still advertise them as ahead.

Common situations: Clusters where bindings were never configured with protocol ssh; removing an up-to-date device from the service while other nodes still need its data; up-to-date devices offline during catch-up.

Related errors

• This host has device ID "%s", but there is no corresponding
• Configuration file specifies cluster peer ("%s", at index "%
• Configuration file specifies cluster peer "%s" more than onc
• This public key is already associated with another user or d
• Analyzing or decrypting SSH keys requires the "ssh-keygen" b

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/86f0541f8dc6e91d. Report an issue: GitHub.