hcengineering/platform · warning

unknown object size

Error message

unknown object size

What it means

In DatalakeClient.putObject, the object size is only computed when the input is a Buffer (stream.length) or a string (Buffer.byteLength). For any other stream type (e.g. a Readable), size stays undefined and the client logs this warning, noting a TODO to implement size calculation for Readable streams. Upload then falls into a different (non-single-request) path since size is unknown.

Source

Thrown at foundations/server/packages/datalake/src/client.ts:238

    }
  }

  async putObject (
    ctx: MeasureContext,
    workspace: WorkspaceUuid,
    objectName: string,
    stream: Readable | Buffer | string,
    params: UploadObjectParams
  ): Promise<ObjectMetadata> {
    let size = params.size
    if (size === undefined) {
      if (Buffer.isBuffer(stream)) {
        size = stream.length
      } else if (typeof stream === 'string') {
        size = Buffer.byteLength(stream)
      } else {
        // TODO: Implement size calculation for Readable streams
        ctx.warn('unknown object size', { workspace, objectName })
      }
    }

    if (size === undefined || size < 64 * 1024 * 1024) {
      return await ctx.with(
        'direct-upload',
        {},
        (ctx) => this.uploadWithFormData(ctx, workspace, objectName, stream, { ...params, size }),
        { workspace, objectName }
      )
    } else {
      return await ctx.with(
        'multipart-upload',
        {},
        (ctx) => this.uploadWithMultipart(ctx, workspace, objectName, stream, { ...params, size }),
        { workspace, objectName }
      )
    }

View on GitHub (pinned to 63e28dc964)

Solutions

1. Read the file fully into a Buffer before uploading (fs.promises.readFile) so the size is known
2. Compute and pass an explicit size option if the client API accepts one
3. For genuinely large/streamed data, accept the multipart path or buffer the stream yourself to determine length
4. Implement/guard the size calculation: track bytes as the Readable is consumed

Example fix

// before
await client.put(ctx, workspace, name, fs.createReadStream(path))
// after
const buf = await fs.promises.readFile(path)
await client.put(ctx, workspace, name, buf)

function ensureSized(input: Buffer | string | Readable): Buffer | string {
  if (Buffer.isBuffer(input) || typeof input === 'string') return input
  throw new Error('datalake putObject requires a Buffer or string (known size)')
}

function hasKnownSize(s: unknown): s is Buffer | string {
  return Buffer.isBuffer(s) || typeof s === 'string'
}

try {
  await client.put(ctx, ws, name, data)
} catch (err) {
  if (err instanceof Error && /unknown object size|size/i.test(err.message)) {
    const buf = await streamToBuffer(data)
    await client.put(ctx, ws, name, buf)
  } else throw err
}

When it happens

Trigger: Calling putObject (directly or via put) with a Node Readable/other stream instead of a Buffer or string, so no known content length is available.

Common situations: Piping a file read stream, HTTP response body, or stream from another source directly into datalake put; refactored code that switched from readFile (Buffer) to createReadStream.

Related errors

• Missing response body
• Bad datalake response:
• Upload failed:
• Failed to start multipart upload
• Server secret not set, datalake storage adapter initialized

AI-assisted analysis of hcengineering/platform@63e28dc964 (2026-08-29). Data as JSON: /api/errors/7ee2ca27a9a1acca. Report an issue: GitHub.