Custom Rich Chat Widgets

This page is the developer guide for adding a new rich chat widget - a YAML definition on the backend plus a React renderer on the frontend. For the catalog of built-in widgets and what each one does for users, see Features → Rich Chat Widgets.

A rich chat widget is a pure display bridge: the YAML tells the LLM what tool to call, the YAML handler returns a confirmation string, and the frontend renders whatever component is registered for that widgetType. No server-side logic in the widget itself - data fetching happens via MCP tools or RTK Query hooks inside the React component.

This page covers LLM-callable display widgets (the model decides whether to show them). Proposal widgets are different: they are server-emitted automatically when a propose_* tool succeeds, so they need no YAML and no show_* tool - only a widgetType and a registered React component (the same render-only mechanism as the built-in proposal-result widget). To add a custom proposal type, follow Custom Proposal Types, not this page.

Step 1 - YAML definition

Drop a file in agent-server/config/rich-chat-widgets/:

# agent-server/config/rich-chat-widgets/vendor-product-preview.yaml
name: vendor_show_preview
description: >-
  Show a preview card for a product in the chat. Call this after
  presenting search results.
widgetType: product-preview
parameters:
  type: object
  properties:
    productId:
      type: number
      description: The product ID
    title:
      type: string
      description: Product name to display
  required: [productId]
handlerTemplate: "Preview card displayed for product #{productId}"

Required fields:

Field	Description
name	Unique tool name the LLM calls.
description	Description the LLM reads to decide when to call this tool.
widgetType	Frontend component key - must match the registry.
parameters	JSON Schema for the tool parameters.
handlerTemplate	Confirmation string sent back to the LLM. {field} placeholders are replaced with arg values.

Step 2 - React renderer

The frontend needs a matching renderer, registered via the Studio UI DI container from your bundle's onInit:

import type { WidgetRendererProps } from '@PimcoreAgentBundle/widgets/widget-renderer-registry'

const registry = container.get<any>('AgentChat/RichChatWidgetRegistry')
registry.register('product-preview', ProductPreviewWidget)

The component receives { data, sessionId }, where data is the tool-call arguments.

Step 3 - Enable in the agent YAML

richChatWidgets:
  - vendor_show_preview
  - pimcore_open_element

Reload:

curl -X POST -H "Authorization: Bearer $AGENT_SERVER_ADMIN_TOKEN" \
  http://localhost/agent-server/api/admin/reload-agents

The widget is now callable by agents that list it.

API reference

YAML contract

• YAML defines: name, description, parameters (JSON Schema), widgetType, handlerTemplate.
• React component receives: { data: <all tool args>, sessionId } - handles rendering and any enrichment.
• Agent YAML enables: add widget name to richChatWidgets.

DI container service

// Service ID
'AgentChat/RichChatWidgetRegistry'

// Methods
registry.register(widgetType: string, component: React.FC<WidgetRendererProps>): void
registry.get(widgetType: string): React.FC<WidgetRendererProps> | undefined
registry.has(widgetType: string): boolean

Node-side helpers

Function	Purpose
loadRichChatWidgets(configDir?)	Load widget YAMLs at startup / on admin reload. Returns count.
resolveRichChatWidgets(names)	Turn an agent's richChatWidgets list into SDK Tool objects. Unknown names are silently skipped.
isRichChatWidget(name)	Is this tool name registered as a widget?
mapRichChatWidgetToEvent(data)	Turn a tool.execution_start event into a widget AgentEvent, or null if not a widget.

For a proposal widget, read this instead

A proposal widget has a stricter contract - three other pieces (propose tool, resolver, proposalType) have to line up with the React component. See Custom Proposal Types.