What causes "get object: failed to get client" in weaviate/weaviate?

Fix BACKUP_S3_ENDPOINT to a bare host[:port] without scheme (see the 'create client' startup error) and restart

get object: failed to get client

Error message

get object: failed to get client

What it means

s3Client.GetObject first re-resolves a client via getClient(ctx), which builds a fresh minio client from per-request X-AWS-ACCESS-KEY / X-AWS-SECRET-KEY / X-AWS-SESSION-TRACK gRPC context headers when present; any failure there is wrapped as 'get object: failed to get client' (modules/backup-s3/client.go:310). The only realistic failure inside getClient is minio.New rejecting the endpoint when constructing the override client, since the default path returns s.client unconditionally.

Source

Thrown at modules/backup-s3/client.go:310

		defer obj.Close()

		var buf bytes.Buffer
		if _, err = io.Copy(&buf, obj); err != nil {
			wrapped := fmt.Errorf("read object: %w", err)
			var s3Err minio.ErrorResponse
			if errors.As(err, &s3Err) && s3Err.StatusCode == http.StatusNotFound {
				return nil, backup.NewErrNotFound(wrapped)
			}
			return nil, wrapped
		}
		return buf.Bytes(), nil
	})
}

func (s *s3Client) GetObject(ctx context.Context, backupID, key, overrideBucket, overridePath string) ([]byte, error) {
	client, err := s.getClient(ctx)
	if err != nil {
		return nil, errors.Wrap(err, "get object: failed to get client")
	}
	bucket, remotePath, err := s.bucketAndPath(backupID, key, overrideBucket, overridePath)
	if err != nil {
		return nil, err
	}

	if err := ctx.Err(); err != nil {
		return nil, backup.NewErrContextExpired(errors.Wrapf(err, "context expired in get object %s", remotePath))
	}

	obj, err := client.GetObject(ctx, bucket, remotePath, minio.GetObjectOptions{})
	if err != nil {
		return nil, backup.NewErrInternal(errors.Wrapf(err, "get object %s", remotePath))
	}

	// Ensure object is closed to prevent connection leaks
	defer obj.Close()

View on GitHub (pinned to 75aa4b6d11)

Solutions

Fix BACKUP_S3_ENDPOINT to a bare host[:port] without scheme (see the 'create client' startup error) and restart
Check the incoming request headers: ensure X-AWS-ACCESS-KEY, X-AWS-SECRET-KEY and X-AWS-SESSION-TOKEN are either all set or all empty — the override client is only built when all three are present
If the endpoint is valid and headers are absent, inspect the wrapped inner error in the logs — getClient otherwise returns the cached client and cannot fail
Retry after correcting configuration; no transient cause exists in this path

Defensive patterns

Strategy: try-catch

Validate before calling

// Before issuing a backup call with header credentials, ensure they are complete:
if (xAccess != "" || xSecret != "" || xToken != "") && !(xAccess != "" && xSecret != "" && xToken != "") {
    return fmt.Errorf("X-AWS-ACCESS-KEY, X-AWS-SECRET-KEY and X-AWS-SESSION-TOKEN must all be set together")
}

Type guard

func isEndpointConfigError(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed to get client")
}

Try / catch

data, err := client.GetObject(ctx, backupID, key, bucket, path)
if err != nil {
    if strings.Contains(err.Error(), "failed to get client") {
        // configuration problem with the S3 endpoint — abort, do not retry
        return fmt.Errorf("backup misconfigured (check BACKUP_S3_ENDPOINT): %w", err)
    }
    return err
}

Prevention

Send all three X-AWS-* headers together or none at all when overriding credentials per request
Treat this error as non-retryable configuration failure; alert rather than retry
Keep endpoint validation identical to the startup check so per-request client building cannot fail

When it happens

Trigger: Calling GetObject (via the backup downloadS3Object flow) with per-request credentials supplied in the request context (all three X-AWS-* headers set) while s.config.Endpoint is malformed — reproducing the minio.New parse error at request time instead of startup time. Note getClient can only error in that per-request-credentials branch.

Common situations: Export/backup requests that pass header-based credentials (multi-tenant key overrides) combined with a bad BACKUP_S3_ENDPOINT that was somehow tolerated at startup; transient config reloads changing the endpoint to an invalid value.

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/7d8646889ecb019e. Report an issue: GitHub.