What causes "cannot upload data to %q at %s (remote path %q): %w" in VictoriaMetrics/VictoriaMetrics?

Check the wrapped Azure error's HTTP status: 403 fix credentials; 404 create/verify the destination container.

cannot upload data to %q at %s (remote path %q): %w

Error message

cannot upload data to %q at %s (remote path %q): %w

What it means

Fires in azremote UploadPart when the Azure block blob UploadStream call fails. Inputs at fault are the part path and remote blob URL; wrapped Azure SDK error usually indicates auth failure, container access level, throttling, or network problems.

Source

Thrown at lib/backup/azremote/azblob.go:288

	if err1 := body.Close(); err1 != nil && err == nil {
		err = err1
	}
	if err != nil {
		return fmt.Errorf("cannot download %q from at %s (remote path %q): %w", p.Path, fs, bc.URL(), err)
	}
	if uint64(n) != p.Size {
		return fmt.Errorf("wrong data size downloaded from %q at %s; got %d bytes; want %d bytes", p.Path, fs, n, p.Size)
	}
	return nil
}

// UploadPart uploads part p from r to fs.
func (fs *FS) UploadPart(p common.Part, r io.Reader) error {
	bc := fs.clientForPart(p)

	_, err := bc.UploadStream(fs.ctx, r, &blockblob.UploadStreamOptions{})
	if err != nil {
		return fmt.Errorf("cannot upload data to %q at %s (remote path %q): %w", p.Path, fs, bc.URL(), err)
	}
	if err := fs.maybeSetMetadata(bc); err != nil {
		return fmt.Errorf("cannot set metadata for %q at %s: %w", p.Path, fs, err)
	}
	return nil
}

func (fs *FS) clientForPart(p common.Part) *blockblob.Client {
	path := p.RemotePath(fs.Dir)

	return fs.clientForPath(path)
}

func (fs *FS) clientForPath(path string) *blockblob.Client {
	bc := fs.client.NewBlockBlobClient(path)
	return bc
}

View on GitHub (pinned to 5079fb58f1)

Solutions

Check the wrapped Azure error's HTTP status: 403 fix credentials; 404 create/verify the destination container.
Verify the storage account configuration (connection string, account name, endpoint).
Retry the upload with a longer-lived context; UploadStream (block blob) retries per-block, but a full retry is safe.
Ensure the source io.Reader is healthy and provides the expected bytes.

Defensive patterns

Strategy: retry

Validate before calling

// verify destination container exists before upload
container := fs.Client().ServiceClient().NewContainerClient(fs.Dir())
if _, err := container.GetProperties(ctx, nil); err != nil {
    return fmt.Errorf("destination container missing: %w", err)
}

Try / catch

if err := fs.UploadPart(p, r); err != nil && strings.Contains(err.Error(), "cannot upload data") {
    var respErr azhttp.ResponseError
    if errors.As(err, &respErr) && respErr.StatusCode == 403 {
        // refresh credentials before retrying
    }
    // otherwise retry with backoff; block-blob upload is idempotent
}

Prevention

Create/verify the destination container before uploading
Use valid, unexpired credentials with write access
Provide a stable io.Reader with the expected byte count
Set context timeouts appropriate for part size and bandwidth

When it happens

Trigger: UploadStream returns an error: 403 authorization failure (bad/expired key or SAS), 404 container-not-found, network interruption mid-stream, context cancellation, quota/account limits, or reader r failing mid-stream and propagating into the upload.

Common situations: Wrong connection string/account key in configuration; destination container deleted; proxy/firewall dropping long uploads; uploading from a reader backed by a failing local source; hitting storage account egress/ingress limits.

Related errors

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/5f60c60f32827526. Report an issue: GitHub.