{"record":{"id":"a69026da958c963e","repo":"dgraph-io/dgraph","slug":"failed-to-read-the-s3-object","errorCode":null,"errorMessage":"Failed to read the s3 object","messagePattern":"Failed to read the s3 object","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"worker/backup_handler.go","lineNumber":316,"sourceCode":"\treturn false\n}\n\nfunc (h *s3Handler) JoinPath(path string) string {\n\treturn filepath.Join(h.bucketName, h.objectPrefix, cleanRelPath(path))\n}\n\nfunc (h *s3Handler) Read(path string) ([]byte, error) {\n\tobjectPath := h.getObjectPath(path)\n\tvar buf bytes.Buffer\n\n\treader, err := h.mc.GetObject(context.Background(), h.bucketName, objectPath, minio.GetObjectOptions{})\n\tif err != nil {\n\t\treturn buf.Bytes(), errors.Wrap(err, \"Failed to read s3 object\")\n\t}\n\tdefer reader.Close()\n\n\tif _, err := buf.ReadFrom(reader); err != nil {\n\t\treturn buf.Bytes(), errors.Wrap(err, \"Failed to read the s3 object\")\n\t}\n\treturn buf.Bytes(), nil\n}\n\nfunc (h *s3Handler) Stream(path string) (io.ReadCloser, error) {\n\tobjectPath := h.getObjectPath(path)\n\treader, err := h.mc.GetObject(context.Background(), h.bucketName, objectPath, minio.GetObjectOptions{})\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn reader, nil\n}\n\nfunc (h *s3Handler) ListPaths(path string) []string {\n\tvar paths []string\n\tpath = h.getObjectPath(path)\n\tfor object := range h.mc.ListObjects(context.Background(), h.bucketName,\n\t\tminio.ListObjectsOptions{Prefix: path, Recursive: true}) {","sourceCodeStart":298,"sourceCodeEnd":334,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/worker/backup_handler.go#L298-L334","documentation":"After GetObject succeeds, s3Handler.Read streams the body into a buffer with buf.ReadFrom; any error during that transfer is wrapped as 'Failed to read the s3 object' (note the 'the'). This indicates the connection broke or the object data was unreadable mid-stream — the data is incomplete/invalid.","triggerScenarios":"buf.ReadFrom(reader) fails mid-transfer: dropped connection to S3, timeout, integrity errors, or the object changed/was deleted while streaming.","commonSituations":"Large backup objects over flaky networks; S3 gateway (minio) restarts mid-read; proxy/load-balancer idle timeouts; expired credentials mid-long-read; truncated objects from failed uploads.","solutions":["Retry Read with exponential backoff — transient network drops are the most common cause.","For large objects, prefer Stream/Range reads or increase client timeouts instead of full-buffer Read.","Verify object integrity (size/ETag) with StatObject; a truncated source object must be re-created from a new backup.","Check credentials validity window for long reads; refresh tokens if applicable.","Inspect the wrapped cause for minio error codes like InternalError or connection reset."],"exampleFix":"// before\nif _, err := buf.ReadFrom(reader); err != nil {\n    return buf.Bytes(), errors.Wrap(err, \"Failed to read the s3 object\")\n}\n// after\nif _, err := buf.ReadFrom(reader); err != nil {\n    return buf.Bytes(), errors.Wrapf(err, \"Failed to read the s3 object: %s (may be partial)\", objectPath)\n}","handlingStrategy":"retry","validationCode":"info, err := h.mc.StatObject(ctx, h.bucketName, h.getObjectPath(path), minio.StatObjectOptions{})\nif err != nil {\n    return fmt.Errorf(\"cannot stat object %s: %v\", path, err)\n}\nlog.Printf(\"expecting %d bytes from %s\", info.Size, path)","typeGuard":null,"tryCatchPattern":"data, err := handler.Read(p)\nif err != nil {\n    if strings.Contains(err.Error(), \"Failed to read the s3 object\") {\n        return retryWithBackoff(5*time.Second, 3, func() error {\n            _, e := handler.Read(p)\n            return e\n        })\n    }\n    return err\n}","preventionTips":["Read over stable networks; avoid huge single reads through proxies with short idle timeouts.","Compare StatObject size with bytes read to detect truncation.","Use atomic multipart uploads so objects are never half-written.","Refresh or extend credential expiry before long operations."],"tags":["s3","minio","network","timeout"],"backgroundTag":"s3-stream-interrupted","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}