{"record":{"id":"06e7586a1d63ca66","repo":"dgraph-io/dgraph","slug":"while-leasing-txn-timestamp-id-v","errorCode":null,"errorMessage":"while leasing txn timestamp. Id: %+v","messagePattern":"while leasing txn timestamp\\. Id: %\\+v","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"dgraph/cmd/zero/tablet.go","lineNumber":191,"sourceCode":"\tif !s.Node.AmLeader() {\n\t\treturn errors.Errorf(\"I am not the Zero leader\")\n\t}\n\tmsg := fmt.Sprintf(\"Going to move predicate: [%v], size: [ondisk: %v, uncompressed: %v]\"+\n\t\t\" from group %d to %d, timeout: %v\\n\", predicate, humanize.IBytes(uint64(tab.OnDiskBytes)),\n\t\thumanize.IBytes(uint64(tab.UncompressedBytes)), srcGroup, dstGroup, timeout)\n\tglog.Info(msg)\n\tspan.SetAttributes(attribute.String(\"tablet\", predicate))\n\tspan.SetStatus(1, msg)\n\n\t// Block all commits on this predicate. Keep them blocked until we return from this function.\n\tunblock := s.blockTablet(predicate)\n\tdefer unblock()\n\n\t// Get a new timestamp, beyond which we are sure that no new txns would be committed for this\n\t// predicate. Source Alpha leader must reach this timestamp before streaming the data.\n\tids, err := s.Timestamps(ctx, &pb.Num{Val: 1})\n\tif err != nil || ids.StartId == 0 {\n\t\treturn errors.Wrapf(err, \"while leasing txn timestamp. Id: %+v\", ids)\n\t}\n\n\t// Get connection to leader of source group.\n\tpl := s.Leader(srcGroup)\n\tif pl == nil {\n\t\treturn errors.Errorf(\"No healthy connection found to leader of group %d\", srcGroup)\n\t}\n\twc := pb.NewWorkerClient(pl.Get())\n\tin := &pb.MovePredicatePayload{\n\t\tPredicate: predicate,\n\t\tSourceGid: srcGroup,\n\t\tDestGid:   dstGroup,\n\t\tTxnTs:     ids.StartId,\n\t}\n\tspan.AddEvent(fmt.Sprintf(\"Move Predicate payload: %+v\", in))\n\tglog.Infof(\"Starting move: %+v\", in)\n\tif _, err := wc.MovePredicate(ctx, in); err != nil {\n\t\treturn errors.Wrapf(err, \"while calling MovePredicate\")","sourceCodeStart":173,"sourceCodeEnd":209,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/dgraph/cmd/zero/tablet.go#L173-L209","documentation":"Before streaming, movePredicate leases one transaction timestamp from Zero's timestamp allocator (s.Timestamps). This timestamp marks the point beyond which no new commits happen for the predicate, so the source Alpha knows where to stop. If leasing fails or returns a zero StartId, the error (if any) is wrapped with 'while leasing txn timestamp'.","triggerScenarios":"s.Timestamps returns an error (context timeout — moveTimeout expired; Zero raft write failing because the node lost leadership or quorum); or ids.StartId == 0 indicating an empty/invalid allocator response, typically when the proposal did not commit.","commonSituations":"Move timeout too small for a slow cluster (large predicate, high load); Zero lost leadership between the quorum check and the timestamp lease; Zero raft group under pressure from many concurrent proposals (other moves, membership changes).","solutions":["Retry the move — the timestamp lease is transactional and safe to redo","Increase moveTimeout if moves of large tablets keep timing out","Verify this Zero is still leader and quorum is healthy via /state and /health","Check Zero logs for raft errors concurrent with the failed lease"],"exampleFix":"// before\n# default move timeout too small for 200GB predicate\n// after\n# restart zero with a larger move window, then retry\nzero --move_timeout=2h --my=... --replicas=...","handlingStrategy":"retry","validationCode":"# ensure zero is leader and healthy before moving\ncurl -sf localhost:6080/state >/dev/null && curl -sf localhost:6080/health >/dev/null \\\n  || { echo \"zero not healthy/leader\"; exit 1; }","typeGuard":null,"tryCatchPattern":"for i in 1 2 3; do\n  resp=$(curl -s \"localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST\")\n  [[ \"$(echo \"$resp\" | jq -r .msg)\" == *\"leasing txn timestamp\"* ]] && { sleep 10; continue; }\n  break\ndone","preventionTips":["Set an adequate --move_timeout on Zero for your largest predicates","Monitor Zero raft health and leadership stability","Avoid issuing many concurrent moves that contend for Zero raft proposals"],"tags":["dgraph","zero","timestamp","raft","timeout"],"backgroundTag":"timestamp-lease-failed","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}