What causes "Failed to read the s3 object" in dgraph-io/dgraph?

Retry Read with exponential backoff — transient network drops are the most common cause.

Failed to read the s3 object

Error message

Failed to read the s3 object

What it means

After GetObject succeeds, s3Handler.Read streams the body into a buffer with buf.ReadFrom; any error during that transfer is wrapped as 'Failed to read the s3 object' (note the 'the'). This indicates the connection broke or the object data was unreadable mid-stream — the data is incomplete/invalid.

Source

Thrown at worker/backup_handler.go:316

	return false
}

func (h *s3Handler) JoinPath(path string) string {
	return filepath.Join(h.bucketName, h.objectPrefix, cleanRelPath(path))
}

func (h *s3Handler) Read(path string) ([]byte, error) {
	objectPath := h.getObjectPath(path)
	var buf bytes.Buffer

	reader, err := h.mc.GetObject(context.Background(), h.bucketName, objectPath, minio.GetObjectOptions{})
	if err != nil {
		return buf.Bytes(), errors.Wrap(err, "Failed to read s3 object")
	}
	defer reader.Close()

	if _, err := buf.ReadFrom(reader); err != nil {
		return buf.Bytes(), errors.Wrap(err, "Failed to read the s3 object")
	}
	return buf.Bytes(), nil
}

func (h *s3Handler) Stream(path string) (io.ReadCloser, error) {
	objectPath := h.getObjectPath(path)
	reader, err := h.mc.GetObject(context.Background(), h.bucketName, objectPath, minio.GetObjectOptions{})
	if err != nil {
		return nil, err
	}
	return reader, nil
}

func (h *s3Handler) ListPaths(path string) []string {
	var paths []string
	path = h.getObjectPath(path)
	for object := range h.mc.ListObjects(context.Background(), h.bucketName,
		minio.ListObjectsOptions{Prefix: path, Recursive: true}) {

View on GitHub (pinned to 759e242be6)

Solutions

Retry Read with exponential backoff — transient network drops are the most common cause.
For large objects, prefer Stream/Range reads or increase client timeouts instead of full-buffer Read.
Verify object integrity (size/ETag) with StatObject; a truncated source object must be re-created from a new backup.
Check credentials validity window for long reads; refresh tokens if applicable.
Inspect the wrapped cause for minio error codes like InternalError or connection reset.

Example fix

// before
if _, err := buf.ReadFrom(reader); err != nil {
    return buf.Bytes(), errors.Wrap(err, "Failed to read the s3 object")
}
// after
if _, err := buf.ReadFrom(reader); err != nil {
    return buf.Bytes(), errors.Wrapf(err, "Failed to read the s3 object: %s (may be partial)", objectPath)
}

Defensive patterns

Strategy: retry

Validate before calling

info, err := h.mc.StatObject(ctx, h.bucketName, h.getObjectPath(path), minio.StatObjectOptions{})
if err != nil {
    return fmt.Errorf("cannot stat object %s: %v", path, err)
}
log.Printf("expecting %d bytes from %s", info.Size, path)

Try / catch

data, err := handler.Read(p)
if err != nil {
    if strings.Contains(err.Error(), "Failed to read the s3 object") {
        return retryWithBackoff(5*time.Second, 3, func() error {
            _, e := handler.Read(p)
            return e
        })
    }
    return err
}

Prevention

Read over stable networks; avoid huge single reads through proxies with short idle timeouts.
Compare StatObject size with bytes read to detect truncation.
Use atomic multipart uploads so objects are never half-written.
Refresh or extend credential expiry before long operations.

When it happens

Trigger: buf.ReadFrom(reader) fails mid-transfer: dropped connection to S3, timeout, integrity errors, or the object changed/was deleted while streaming.

Common situations: Large backup objects over flaky networks; S3 gateway (minio) restarts mid-read; proxy/load-balancer idle timeouts; expired credentials mid-long-read; truncated objects from failed uploads.

Related errors

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/a69026da958c963e. Report an issue: GitHub.