twentyhq/twenty · critical · Error

Failed to backfill FIELDS widgets for workspace ${workspaceI

Error message

Failed to backfill FIELDS widgets for workspace ${workspaceId}

What it means

Thrown by the 2.9 workspace upgrade command that backfills FIELDS-type page layout widgets to set newFieldDefaultVisibility: true. It fires after validateBuildAndRunLegacyWorkspaceMigration returns status === 'fail' for a per-application pageLayoutWidget update batch. Each standard application is processed in its own loop iteration; the throw is per-application but reports the workspace.

Source

Thrown at packages/twenty-server/src/database/commands/upgrade-version-command/2-9/2-9-workspace-command-1799000030000-backfill-fields-widget-new-field-default-visibility.command.ts:124

        await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunLegacyWorkspaceMigration(
          {
            allFlatEntityOperationByMetadataName: {
              pageLayoutWidget: {
                flatEntityToCreate: [],
                flatEntityToDelete: [],
                flatEntityToUpdate: updatedWidgets,
              },
            },
            workspaceId,
            applicationUniversalIdentifier,
          },
        );

      if (result.status === 'fail') {
        this.logger.error(
          `Failed to backfill FIELDS widgets for application ${applicationUniversalIdentifier} in workspace ${workspaceId}:\n${JSON.stringify(result, null, 2)}`,
        );
        throw new Error(
          `Failed to backfill FIELDS widgets for workspace ${workspaceId}`,
        );
      }

      this.logger.log(
        `Backfilled ${updatedWidgets.length} FIELDS widget(s) for application ${applicationUniversalIdentifier} in workspace ${workspaceId}`,
      );
    }
  }
}

View on GitHub (pinned to 1f5dd2bbd2)

Solutions

1. Read the preceding log: `Failed to backfill FIELDS widgets for application <id> in workspace <id>:` plus the JSON result — the result object carries the exact field-level failure.
2. Query the failing workspace's pageLayoutWidget rows for that applicationUniversalIdentifier (filter type = FIELDS) and compare each row's shape against the builder's expectations (presence of universalConfiguration, view fields).
3. Fix or remove the malformed widget row, then re-run the upgrade command for that workspace.
4. If many applications fail identically, suspect a schema drift — run any pending fast instance commands before the workspace command.
5. Confirm newFieldDefaultVisibility is a recognized field in the pageLayoutWidget entity definition for this server version.

Example fix

// before
throw new Error(`Failed to backfill FIELDS widgets for workspace ${workspaceId}`);
// after — surface the application id and the structured failure so triage does not require cross-referencing logs
type BuildFail = { status: 'fail'; error?: string; message?: string };
const fail = result as BuildFail;
throw new Error(
  `Failed to backfill FIELDS widgets for application ${applicationUniversalIdentifier} ` +
  `in workspace ${workspaceId}: ${fail.error ?? fail.message ?? JSON.stringify(result)}`,
);

// Pre-check FIELDS widgets for the application before running the upgrade.
const widgets = await dataSource.query(
  `SELECT id, "applicationUniversalIdentifier", "universalConfiguration" FROM ${workspaceSchema}.pageLayoutWidget WHERE type = 'FIELDS'`,
);
const bad = widgets.filter((w: any) => w.universalConfiguration == null);
if (bad.length) {
  throw new Error(`FIELDS widgets with null universalConfiguration: ${bad.map((w: any) => w.id).join(', ')}`);
}

type BuildResult =
  | { status: 'success' }
  | { status: 'fail'; error?: string; message?: string };
function isFail(r: BuildResult): r is Extract<BuildResult, { status: 'fail' }> {
  return r.status === 'fail';
}

try {
  await backfillCommand.run();
} catch (err) {
  if (err instanceof Error && err.message.startsWith('Failed to backfill FIELDS widgets')) {
    failed.push({ workspaceId, err });
    // do not abort the whole run; continue other applications/workspaces
  } else throw err;
}

When it happens

Trigger: Upgrading to 2.9 on a workspace where one or more FIELDS widgets have a pageLayoutWidget record whose shape does not match what the migration builder expects — e.g. a widget missing the viewFilter/viewGrouping/viewSort metadata the builder reads, or a universalConfiguration that fails JSON validation. Also fires when the workspace's metadata schema for pageLayoutWidget lags behind the code version expected by this command.

Common situations: Self-hosted instances that skipped the 2.8 step, workspaces with custom/edited views whose widget JSON was hand-modified, or a DB that was partially restored. The preceding error log line names the failing applicationUniversalIdentifier and dumps the full result JSON, which holds the validation message.

Related errors

• Failed to gate default command menu items by permission flag
• Failed to update search command menu item labels for workspa
• Failed to add Send Email record-selection commands for works
• Failed to backfill standard skills for workspace ${workspace
• Failed to fix merge command menu item expression for workspa

AI-assisted analysis of twentyhq/twenty@1f5dd2bbd2 (2026-08-12). Data as JSON: /api/errors/794aff6574f78d1b. Report an issue: GitHub.