{"record":{"id":"7d8646889ecb019e","repo":"weaviate/weaviate","slug":"get-object-failed-to-get-client","errorCode":null,"errorMessage":"get object: failed to get client","messagePattern":"get object: failed to get client","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"modules/backup-s3/client.go","lineNumber":310,"sourceCode":"\t\tdefer obj.Close()\n\n\t\tvar buf bytes.Buffer\n\t\tif _, err = io.Copy(&buf, obj); err != nil {\n\t\t\twrapped := fmt.Errorf(\"read object: %w\", err)\n\t\t\tvar s3Err minio.ErrorResponse\n\t\t\tif errors.As(err, &s3Err) && s3Err.StatusCode == http.StatusNotFound {\n\t\t\t\treturn nil, backup.NewErrNotFound(wrapped)\n\t\t\t}\n\t\t\treturn nil, wrapped\n\t\t}\n\t\treturn buf.Bytes(), nil\n\t})\n}\n\nfunc (s *s3Client) GetObject(ctx context.Context, backupID, key, overrideBucket, overridePath string) ([]byte, error) {\n\tclient, err := s.getClient(ctx)\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"get object: failed to get client\")\n\t}\n\tbucket, remotePath, err := s.bucketAndPath(backupID, key, overrideBucket, overridePath)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif err := ctx.Err(); err != nil {\n\t\treturn nil, backup.NewErrContextExpired(errors.Wrapf(err, \"context expired in get object %s\", remotePath))\n\t}\n\n\tobj, err := client.GetObject(ctx, bucket, remotePath, minio.GetObjectOptions{})\n\tif err != nil {\n\t\treturn nil, backup.NewErrInternal(errors.Wrapf(err, \"get object %s\", remotePath))\n\t}\n\n\t// Ensure object is closed to prevent connection leaks\n\tdefer obj.Close()\n","sourceCodeStart":292,"sourceCodeEnd":328,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/modules/backup-s3/client.go#L292-L328","documentation":"s3Client.GetObject first re-resolves a client via getClient(ctx), which builds a fresh minio client from per-request X-AWS-ACCESS-KEY / X-AWS-SECRET-KEY / X-AWS-SESSION-TRACK gRPC context headers when present; any failure there is wrapped as 'get object: failed to get client' (modules/backup-s3/client.go:310). The only realistic failure inside getClient is minio.New rejecting the endpoint when constructing the override client, since the default path returns s.client unconditionally.","triggerScenarios":"Calling GetObject (via the backup downloadS3Object flow) with per-request credentials supplied in the request context (all three X-AWS-* headers set) while s.config.Endpoint is malformed — reproducing the minio.New parse error at request time instead of startup time. Note getClient can only error in that per-request-credentials branch.","commonSituations":"Export/backup requests that pass header-based credentials (multi-tenant key overrides) combined with a bad BACKUP_S3_ENDPOINT that was somehow tolerated at startup; transient config reloads changing the endpoint to an invalid value.","solutions":["Fix BACKUP_S3_ENDPOINT to a bare host[:port] without scheme (see the 'create client' startup error) and restart","Check the incoming request headers: ensure X-AWS-ACCESS-KEY, X-AWS-SECRET-KEY and X-AWS-SESSION-TOKEN are either all set or all empty — the override client is only built when all three are present","If the endpoint is valid and headers are absent, inspect the wrapped inner error in the logs — getClient otherwise returns the cached client and cannot fail","Retry after correcting configuration; no transient cause exists in this path"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before issuing a backup call with header credentials, ensure they are complete:\nif (xAccess != \"\" || xSecret != \"\" || xToken != \"\") && !(xAccess != \"\" && xSecret != \"\" && xToken != \"\") {\n    return fmt.Errorf(\"X-AWS-ACCESS-KEY, X-AWS-SECRET-KEY and X-AWS-SESSION-TOKEN must all be set together\")\n}","typeGuard":"func isEndpointConfigError(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to get client\")\n}","tryCatchPattern":"data, err := client.GetObject(ctx, backupID, key, bucket, path)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to get client\") {\n        // configuration problem with the S3 endpoint — abort, do not retry\n        return fmt.Errorf(\"backup misconfigured (check BACKUP_S3_ENDPOINT): %w\", err)\n    }\n    return err\n}","preventionTips":["Send all three X-AWS-* headers together or none at all when overriding credentials per request","Treat this error as non-retryable configuration failure; alert rather than retry","Keep endpoint validation identical to the startup check so per-request client building cannot fail"],"tags":["s3","minio","configuration","backup"],"backgroundTag":"invalid-endpoint","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}