What causes "Failed to read s3 object" in dgraph-io/dgraph?

Check the wrapped cause for minio's error type (BucketDoesNotExist, ObjectDoesNotExist, auth, connection).

Failed to read s3 object

Error message

Failed to read s3 object

What it means

s3Handler.Read wraps minio.GetObject client-side failures with 'Failed to read s3 object'. This fires when obtaining the object reader itself fails (client/config error); errors during streaming the body produce the sibling 'Failed to read the s3 object' message instead.

Source

Thrown at worker/backup_handler.go:311

	}
	errResponse := minio.ToErrorResponse(err)
	if errResponse.Code != "NoSuchKey" {
		glog.Errorf("Failed to verify object existence: %v", err)
	}
	return false
}

func (h *s3Handler) JoinPath(path string) string {
	return filepath.Join(h.bucketName, h.objectPrefix, cleanRelPath(path))
}

func (h *s3Handler) Read(path string) ([]byte, error) {
	objectPath := h.getObjectPath(path)
	var buf bytes.Buffer

	reader, err := h.mc.GetObject(context.Background(), h.bucketName, objectPath, minio.GetObjectOptions{})
	if err != nil {
		return buf.Bytes(), errors.Wrap(err, "Failed to read s3 object")
	}
	defer reader.Close()

	if _, err := buf.ReadFrom(reader); err != nil {
		return buf.Bytes(), errors.Wrap(err, "Failed to read the s3 object")
	}
	return buf.Bytes(), nil
}

func (h *s3Handler) Stream(path string) (io.ReadCloser, error) {
	objectPath := h.getObjectPath(path)
	reader, err := h.mc.GetObject(context.Background(), h.bucketName, objectPath, minio.GetObjectOptions{})
	if err != nil {
		return nil, err
	}
	return reader, nil
}

View on GitHub (pinned to 759e242be6)

Solutions

Check the wrapped cause for minio's error type (BucketDoesNotExist, ObjectDoesNotExist, auth, connection).
Verify credentials (access/secret, session token) and endpoint/TLS settings.
Confirm the object exists: mc ls / head-object on bucket+objectPath (check objectPrefix).
Test network reachability to the endpoint (curl/DNS) and correct region/virtual-host settings.
Re-run the operation once connectivity/auth is fixed — reads are safe to retry.

Example fix

// before
reader, err := h.mc.GetObject(ctx, h.bucketName, objectPath, minio.GetObjectOptions{})
if err != nil {
    return buf.Bytes(), errors.Wrap(err, "Failed to read s3 object")
}
// after
reader, err := h.mc.GetObject(ctx, h.bucketName, objectPath, minio.GetObjectOptions{})
if err != nil {
    resp := minio.ToErrorResponse(err)
    return buf.Bytes(), errors.Wrapf(err, "Failed to read s3 object (code=%s)", resp.Code)
}

Defensive patterns

Strategy: retry

Validate before calling

ok, err := h.mc.BucketExists(ctx, h.bucketName)
if err != nil || !ok {
    return fmt.Errorf("bucket %s unreachable or missing: %v", h.bucketName, err)
}
if _, err := h.mc.StatObject(ctx, h.bucketName, h.getObjectPath(path), minio.StatObjectOptions{}); err != nil {
    return fmt.Errorf("object %s missing: %v", path, err)
}

Try / catch

data, err := handler.Read(p)
if err != nil {
    if strings.Contains(err.Error(), "Failed to read s3 object") {
        return retryWithBackoff(3, func() error { _, e := handler.Read(p); return e })
    }
    return err
}

Prevention

Validate S3 credentials and bucket existence at startup.
Log resolved object paths (prefix + path) to catch prefix misconfig.
Use short retry with backoff for transient network errors.
Keep lifecycle rules from deleting active backup objects.

When it happens

Trigger: minio.GetObject returns an error before any read: invalid bucket/object path, bad credentials/endpoint configuration, network failure to the S3 endpoint, or wrong-region bucket — while calling s3Handler.Read.

Common situations: Wrong minio endpoint or TLS config; deleted or non-existent backup object; IAM/key revoked; DNS/network outage to the S3 endpoint; object name mismatch due to prefix misconfiguration.

Related errors

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/c6a17e17a5b1f535. Report an issue: GitHub.