{"record":{"id":"12887eea78fbad81","repo":"juicedata/juicefs","slug":"object-s-was-removed-during-sync","errorCode":null,"errorMessage":"object %s was removed during sync","messagePattern":"object (.+?) was removed during sync","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/sync/sync.go","lineNumber":1258,"sourceCode":"\t\tequal := cur.Size() == obj.Size()\n\t\tif equal && !cur.Mtime().Equal(obj.Mtime()) {\n\t\t\t// Head of an object may not return the millisecond part of mtime as List\n\t\t\tequal = cur.Mtime().Unix() == obj.Mtime().Unix() && cur.Mtime().UnixMilli()%1000 == 0\n\t\t}\n\t\tif !equal {\n\t\t\treturn fmt.Errorf(\"%s changed during sync. Original: size=%d, mtime=%s; Current: size=%d, mtime=%s\",\n\t\t\t\tcur.Key(), obj.Size(), obj.Mtime(), cur.Size(), cur.Mtime())\n\t\t}\n\t\tif dstObj, err := dst.Head(ctx, key); err == nil {\n\t\t\tif cur.Size() != dstObj.Size() {\n\t\t\t\treturn fmt.Errorf(\"copied %s size mismatch: original=%d, current=%d\", key, obj.Size(), dstObj.Size())\n\t\t\t}\n\t\t\treturn nil\n\t\t} else {\n\t\t\treturn fmt.Errorf(\"check %s in %s: %s\", key, dst, err)\n\t\t}\n\t} else if errors.Is(err, os.ErrNotExist) {\n\t\treturn fmt.Errorf(\"object %s was removed during sync\", key)\n\t} else {\n\t\treturn fmt.Errorf(\"check %s in %s: %s\", key, src, err)\n\t}\n}\n\nfunc copyLink(src object.ObjectStorage, dst object.ObjectStorage, key string) error {\n\tvar p string\n\tvar err error\n\tif p, err = src.(object.SupportSymlink).Readlink(key); err != nil {\n\t\treturn err\n\t}\n\treturn try(3, func() (err error) {\n\t\t// TODO: use relative path based on option\n\t\tif err := dst.(object.SupportSymlink).Symlink(p, key); err != nil {\n\t\t\tif info, err := dst.Head(ctx, key); err == nil && info.IsSymlink() {\n\t\t\t\tif cPath, err2 := dst.(object.SupportSymlink).Readlink(key); err2 == nil && p == cPath {\n\t\t\t\t\treturn nil\n\t\t\t\t}","sourceCodeStart":1240,"sourceCodeEnd":1276,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/sync/sync.go#L1240-L1276","documentation":"Raised by checkChange when the SOURCE object no longer exists at verification time: errors.Is(err, os.ErrNotExist) from src.Head. The object was listed at the start of the sync but deleted before the check ran, so a consistent copy can no longer be guaranteed.","triggerScenarios":"sync lists objects, copies one, then calls checkChange; src.Head returns os.ErrNotExist because the object was deleted concurrently during the sync run.","commonSituations":"Log rotation/cleanup jobs deleting files from the source while sync runs; lifecycle expiration rules on the source bucket; a second sync or user deleting keys concurrently; syncing directories with ephemeral/temp files.","solutions":["Rerun sync — the next listing won't include the deleted object; this error for a genuinely deleted object is usually harmless and self-healing.","Exclude ephemeral prefixes (tmp/, *.log rotation targets) with --exclude to avoid racing deleters.","Quiesce or reschedule cleanup jobs and source lifecycle rules during the sync window.","If deletions should propagate to the destination, run sync with a delete-propagation option instead of treating it as an error."],"exampleFix":"// exclude ephemeral files that get deleted mid-sync\njuicefs sync src://bucket dst://bucket --exclude 'tmp/*' --exclude '*.tmp'","handlingStrategy":"try-catch","validationCode":"// detect racy sources up front: list twice and compare presence\na, _ := listKeys(src, prefix)\nb, _ := listKeys(src, prefix)\nif len(diff(a, b)) > 0 { log.Print(\"source keys are being deleted concurrently\") }","typeGuard":null,"tryCatchPattern":"if err := sync(...); err != nil {\n  if strings.Contains(err.Error(), \"was removed during sync\") {\n    logger.Warnf(\"skipping object deleted mid-sync: %v\", err)\n    return nil // benign for append/ephemeral sources\n  }\n  return err\n}","preventionTips":["Exclude ephemeral/tmp prefixes with --exclude.","Pause cleanup jobs and lifecycle expiration during the sync window.","Treat single occurrences as benign; rerun sync to get a clean pass.","Use delete-propagation semantics explicitly if deletions should be mirrored."],"tags":["sync","race","source-deleted"],"backgroundTag":"file-not-found","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}