weaviate/weaviate · error

write metadata after %d attempts: %w

Error message

write metadata after %d attempts: %w

What it means

The terminal failure of writeExportMetadata: after all 3 attempts of backend.Write failed (and the context did not expire mid-wait), the last backend error is returned wrapped as "write metadata after 3 attempts: <err>". Any export start, status update, cancel, or promotion that needs to persist metadata fails with this.

Source

Thrown at usecases/export/scheduler.go:864

	const maxRetries = 3
	for attempt := range maxRetries {
		_, err = backend.Write(ctx, exportID, exportMetadataFile, bucket, path, newBytesReadCloser(data))
		if err == nil {
			return nil
		}
		if attempt < maxRetries-1 {
			logger.WithField("action", "export_write_metadata").
				WithField("export_id", exportID).
				Warnf("metadata write attempt %d failed, retrying: %v", attempt+1, err)
			select {
			case <-ctx.Done():
				return fmt.Errorf("write metadata aborted after %d attempts: %w", attempt+1, ctx.Err())
			case <-time.After(500 * time.Millisecond):
			}
		}
	}
	return fmt.Errorf("write metadata after %d attempts: %w", maxRetries, err)
}

// readNodeStatus reads and unmarshals a node's status file from the storage backend.
func readNodeStatus(ctx context.Context, backend modulecapabilities.BackupBackend, exportID, bucket, path, nodeName string) (*NodeStatus, error) {
	key := fmt.Sprintf("node_%s_status.json", nodeName)
	data, err := backend.GetObject(ctx, exportID, key, bucket, path)
	if err != nil {
		return nil, fmt.Errorf("get node status: %w", err)
	}

	var status NodeStatus
	if err := json.Unmarshal(data, &status); err != nil {
		return nil, fmt.Errorf("unmarshal node status: %w", err)
	}

	return &status, nil
}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Fix the wrapped backend error: for 403/AccessDenied check credentials/IAM, for 404/NoSuchBucket check bucket name and path prefix configuration.
2. Validate the backend config by running a backup to the same bucket/path — it shares the BackupBackend.
3. Check clock skew if using signature-based auth (S3), which causes fast auth rejections.
4. Retry the operation once the backend is writable; if this occurred during startExport the nodes were aborted and no partial export remains.

// Verify backend config (bucket, path, credentials) before any export/cancel.
if err := backendWriteProbe(bucket, path); err != nil {
    return fmt.Errorf("backend misconfigured: %w", err)
}

if err := startExport(...); err != nil {
    if strings.Contains(err.Error(), "write metadata after 3 attempts") {
        // persistent backend rejection: fix credentials/bucket/permissions, then retry
        log.Printf("backend persistently rejecting metadata writes: %v", err)
    }
    return err
}

When it happens

Trigger: writeExportMetadata: three consecutive backend.Write calls returned errors without the 10s context expiring between retries — e.g. persistent auth failure, missing bucket, or immediate backend rejections.

Common situations: Wrong or expired object-storage credentials, bucket/path does not exist or lacks write permission, backend misconfiguration after a config change, or the storage provider returning fast errors (403/404).

Related errors

• failed to write export metadata: %w
• %w (follow-up: failed to persist failure metadata: %w)
• write metadata aborted after %d attempts: %w
• shard %s of class %s has only empty assigned nodes
• initialize backend: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/3ac92690d8cd18d0. Report an issue: GitHub.