{"record":{"id":"3ac92690d8cd18d0","repo":"weaviate/weaviate","slug":"write-metadata-after-d-attempts-w","errorCode":null,"errorMessage":"write metadata after %d attempts: %w","messagePattern":"write metadata after (.+?) attempts: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"usecases/export/scheduler.go","lineNumber":864,"sourceCode":"\n\tconst maxRetries = 3\n\tfor attempt := range maxRetries {\n\t\t_, err = backend.Write(ctx, exportID, exportMetadataFile, bucket, path, newBytesReadCloser(data))\n\t\tif err == nil {\n\t\t\treturn nil\n\t\t}\n\t\tif attempt < maxRetries-1 {\n\t\t\tlogger.WithField(\"action\", \"export_write_metadata\").\n\t\t\t\tWithField(\"export_id\", exportID).\n\t\t\t\tWarnf(\"metadata write attempt %d failed, retrying: %v\", attempt+1, err)\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn fmt.Errorf(\"write metadata aborted after %d attempts: %w\", attempt+1, ctx.Err())\n\t\t\tcase <-time.After(500 * time.Millisecond):\n\t\t\t}\n\t\t}\n\t}\n\treturn fmt.Errorf(\"write metadata after %d attempts: %w\", maxRetries, err)\n}\n\n// readNodeStatus reads and unmarshals a node's status file from the storage backend.\nfunc readNodeStatus(ctx context.Context, backend modulecapabilities.BackupBackend, exportID, bucket, path, nodeName string) (*NodeStatus, error) {\n\tkey := fmt.Sprintf(\"node_%s_status.json\", nodeName)\n\tdata, err := backend.GetObject(ctx, exportID, key, bucket, path)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"get node status: %w\", err)\n\t}\n\n\tvar status NodeStatus\n\tif err := json.Unmarshal(data, &status); err != nil {\n\t\treturn nil, fmt.Errorf(\"unmarshal node status: %w\", err)\n\t}\n\n\treturn &status, nil\n}\n","sourceCodeStart":846,"sourceCodeEnd":882,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/usecases/export/scheduler.go#L846-L882","documentation":"The terminal failure of writeExportMetadata: after all 3 attempts of backend.Write failed (and the context did not expire mid-wait), the last backend error is returned wrapped as \"write metadata after 3 attempts: <err>\". Any export start, status update, cancel, or promotion that needs to persist metadata fails with this.","triggerScenarios":"writeExportMetadata: three consecutive backend.Write calls returned errors without the 10s context expiring between retries — e.g. persistent auth failure, missing bucket, or immediate backend rejections.","commonSituations":"Wrong or expired object-storage credentials, bucket/path does not exist or lacks write permission, backend misconfiguration after a config change, or the storage provider returning fast errors (403/404).","solutions":["Fix the wrapped backend error: for 403/AccessDenied check credentials/IAM, for 404/NoSuchBucket check bucket name and path prefix configuration.","Validate the backend config by running a backup to the same bucket/path — it shares the BackupBackend.","Check clock skew if using signature-based auth (S3), which causes fast auth rejections.","Retry the operation once the backend is writable; if this occurred during startExport the nodes were aborted and no partial export remains."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify backend config (bucket, path, credentials) before any export/cancel.\nif err := backendWriteProbe(bucket, path); err != nil {\n    return fmt.Errorf(\"backend misconfigured: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := startExport(...); err != nil {\n    if strings.Contains(err.Error(), \"write metadata after 3 attempts\") {\n        // persistent backend rejection: fix credentials/bucket/permissions, then retry\n        log.Printf(\"backend persistently rejecting metadata writes: %v\", err)\n    }\n    return err\n}","preventionTips":["Validate storage credentials and bucket write permissions whenever config changes.","Watch for fast repeated backend errors (403/404) — they exhaust retries immediately.","Keep NTP/clock skew in check for S3 signature auth; skew causes instant auth failures.","Test the backend with a backup run before scheduling cluster-wide exports."],"tags":["export","storage-backend","retry-exhausted","metadata","s3"],"backgroundTag":"metadata-write-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}