{"record":{"id":"86f0541f8dc6e91d","repo":"phacility/phabricator","slug":"leader-lost-no-up-to-date-nodes-in-repository-clu","errorCode":null,"errorMessage":"Leader lost: no up-to-date nodes in repository cluster are fetchable.","messagePattern":"Leader lost: no up-to-date nodes in repository cluster are fetchable\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php","lineNumber":715,"sourceCode":"    $fetchable = array();\n    foreach ($bindings as $binding) {\n      // We can't fetch from nodes which don't have the newest version.\n      $device_phid = $binding->getDevicePHID();\n      if (empty($device_map[$device_phid])) {\n        continue;\n      }\n\n      // TODO: For now, only fetch over SSH. We could support fetching over\n      // HTTP eventually.\n      if ($binding->getAlmanacPropertyValue('protocol') != 'ssh') {\n        continue;\n      }\n\n      $fetchable[] = $binding;\n    }\n\n    if (!$fetchable) {\n      throw new Exception(\n        pht(\n          'Leader lost: no up-to-date nodes in repository cluster are '.\n          'fetchable.'));\n    }\n\n    // If we can synchronize from multiple sources, choose one at random.\n    shuffle($fetchable);\n\n    $caught = null;\n    foreach ($fetchable as $binding) {\n      try {\n        $this->synchronizeWorkingCopyFromBinding(\n          $binding,\n          $local_version,\n          $remote_version);\n        $caught = null;\n        break;\n      } catch (Exception $ex) {","sourceCodeStart":697,"sourceCodeEnd":733,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php#L697-L733","documentation":"This device is behind and needs to fetch from an up-to-date node. Version rows say up-to-date devices exist, but after filtering (bindings whose device is not in the current device set, and bindings whose protocol is not 'ssh', since the TODO notes only SSH fetching is supported) nothing is fetchable. The engine treats the leader as lost and throws rather than fetch stale data.","triggerScenarios":"Every up-to-date device is bound with a protocol other than ssh (for example all bindings left at http), or the up-to-date devices' bindings are disabled/removed while their version rows still advertise them as ahead.","commonSituations":"Clusters where bindings were never configured with protocol ssh; removing an up-to-date device from the service while other nodes still need its data; up-to-date devices offline during catch-up.","solutions":["Give at least one up-to-date device an active binding with protocol 'ssh' and confirm that host is reachable from the lagging device","Re-bind or bring back online the up-to-date device that holds the newest version","If no up-to-date copy is recoverable, this is data loss: follow the Cluster: Repositories documentation to rebuild from the best surviving copy and reset the version rows deliberately"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before demoting or removing nodes, assert an up-to-date ssh binding remains\n$fetchable = array();\nforeach ($service->getActiveBindings() as $binding) {\n  if ($binding->getAlmanacPropertyValue('protocol') === 'ssh'\n      && isset($up_to_date[$binding->getDevicePHID()])) {\n    $fetchable[] = $binding;\n  }\n}\nif (!$fetchable) {\n  // refuse the maintenance action: it would strand lagging devices\n}","typeGuard":"function hasFetchableUpToDateBinding(array $bindings, array $up_to_date_phids) {\n  foreach ($bindings as $binding) {\n    if ($binding->getAlmanacPropertyValue('protocol') === 'ssh'\n        && in_array($binding->getDevicePHID(), $up_to_date_phids, true)) {\n      return true;\n    }\n  }\n  return false;\n}","tryCatchPattern":"try {\n  $engine->synchronizeWorkingCopyFromDevices($phids, $local, $remote);\n} catch (Exception $ex) {\n  // data availability problem: alert operators, do not silently fall back to stale data\n}","preventionTips":["Keep at least two ssh-protocol bindings per repository across different devices","Never configure all bindings as http-only; ssh is the only supported fetch protocol","Rehearse node failover so up-to-date replicas are never accidentally all removed"],"tags":["cluster","synchronization","ssh","availability","phabricator"],"backgroundTag":"no-healthy-replicas","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}