dgraph-io/dgraph · error

cannot create tablet for restored predicate %s

Error message

cannot create tablet for restored predicate %s

What it means

Wrapped by handleRestoreProposal (worker/online_restore.go:332). For each predicate in the backup's group entry, restore calls groups().ForceTablet(pred) to force the tablet for that predicate onto this group in Zero; failure means Zero refused or failed to create/move the tablet (wrapped error carries the Zero-side cause).

Source

Thrown at worker/online_restore.go:332

		// predicate's group. Registering them here would create an inconsistent
		// state that causes backup manifest duplication and may trigger futile
		// rebalancing attempts.
		//
		// Skipping ForceTablet is safe because the restore mapper decides which
		// data keys to restore based on the manifest's predicate set (predSet),
		// not on Zero's tablet assignments. The supporting predicate data will
		// still be restored to the correct Alpha's Badger store.
		if strings.HasSuffix(pred, hnsw.VecEntry) ||
			strings.HasSuffix(pred, hnsw.VecKeyword) ||
			strings.HasSuffix(pred, hnsw.VecDead) {
			continue
		}

		// Force the tablet to be moved to this group, even
		// if it's currently being served by another group.
		tablet, err := groups().ForceTablet(pred)
		if err != nil {
			return errors.Wrapf(err, "cannot create tablet for restored predicate %s", pred)
		}
		if tablet.GetGroupId() != req.GroupId {
			return errors.Errorf("cannot assign tablet for pred %s to group %d", pred, req.GroupId)
		}
	}

	mapDir, err := os.MkdirTemp(x.WorkerConfig.TmpDir, "restore-map")
	x.Check(err)
	defer func() {
		if err := os.RemoveAll(mapDir); err != nil {
			glog.Warningf("Error removing temp restore-map dir: %v", err)
		}
	}()
	glog.Infof("Created temporary map directory: %s\n", mapDir)

	// Map the backup.
	mapRes, err := RunMapper(req, mapDir)
	if err != nil {

View on GitHub (pinned to 759e242be6)

Solutions

  1. Check Zero leader health (all Zero replicas up, a leader elected) and retry the restore.
  2. Ensure no concurrent moveTablet operations or scaling events run during restore; quiesce the cluster before restoring.
  3. Inspect the wrapped Zero error in Alpha logs for the exact ForceTablet failure and address it (e.g. membership conflict).
  4. Retry restore after the cluster membership is stable.

Example fix

# before: restore during tablet move / zero election fails
# after: verify stable zero leader, then restore
curl localhost:6080/state | jq '.zeros[].leader'
curl -X POST localhost:8080/restore -d '{"location": "s3://bucket/backup", "groupId": 1}'
Defensive patterns

Strategy: retry

Validate before calling

// Pre-check Zero leader stability before restore
resp, _ := http.Get("http://zero:6080/state")
// confirm a leader exists and no moveTablet operations pending

Try / catch

err := restore(ctx, req)
if err != nil && strings.Contains(err.Error(), "cannot create tablet for restored predicate") {
    // wait for Zero leader + membership stability, then retry
    time.Sleep(time.Minute)
    err = restore(ctx, req)
}

Prevention

When it happens

Trigger: Zero unreachable or not the leader when the restore proposal applies; predicate already being moved by another rebalancer/moveTablet operation; Zero returns an error for a tablet that conflicts with current membership state.

Common situations: Restore executed while Zero is electing a leader; concurrent moveTablet operations or a tablet-snapshot conflict; membership churn (Alpha joining/leaving) during restore; stale Zero state after a crash.

Related errors


AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/b1566be9f0704ad2. Report an issue: GitHub.