Model Evaluation

We evaluated a range of models against the bundle's real agent tasks to ground the guidance in Model Requirements. This page documents how the evaluation works and the measured results.

Methodology

The eval tool

The agent-server ships a CLI evaluation harness under agent-server/eval. It drives real agent sessions against the live PHP backend and records full metrics per run.

cd dev/pimcore/pimcore-agent-bundle/agent-server
npm run eval -- list                       # list available tasks and agents
npm run eval -- run <taskId> --batch <b>   # run a task, tagged into a batch
npm run eval -- summary --batch <b>        # aggregate a batch into a metrics table

Other subcommands (prompt, ask-reply, proposal, report, status) drive follow-up turns, answer questions, resolve proposals, and emit per-session reports.

Claude Code as the operator

Each task is run with Claude Code acting as the operator that drives the conversation the way a real user would: it sends the task prompt, issues refinement messages when the first attempt is incomplete, and approves, refines, or rejects the agent's proposals - then judges the outcome and records a pass/fail verdict with notes.

Mechanical "continue or stop?" checkpoints are handled by an autopilot so operator latency does not distort timings: it auto-answers continue until the task completes, a real question surfaces (hands back to the operator), the per-task tool-call ceiling is hit, or a maximum number of continue round-trips is reached.

The tasks

The suite has 18 tasks spanning the capabilities agents are used for. Each runs against a sample agent definition shipped with the bundle which can be run against our demo dataset.

#	Task	What it exercises
1	smoke-list-classes	Basic tool wiring - call the read tool and list the available data-object classes instead of answering from memory.
2	reason-data-model	Read and explain the data model: the Manufacturer ↔ Car relation plus the fields the Car class itself holds.
3	load-red-cars	Query cars filtered by color and present them as interactive object-preview widgets (not a table or prose list).
4	load-bmw-cars-table	Query cars by manufacturer and render them in a data-table widget with sensible columns.
5	load-aston-martin-assets	Find image assets for a manufacturer and present them through asset-preview widgets.
6	summarize-car	Read a single car and summarise its key fields, emitting an element-link widget to the object.
7	update-car-object	Propose a multi-field update (bodystyle, class, year, colors, images, dimensions) as one approved change-set.
8	update-accessorypart-object	Edit nested classification-store data - round numeric dimensions and change a typed value.
9	update-event-object	Multi-language update (title/description across locales) plus tags, location, status, and computed dates.
10	update-multiple-cars	Produce one separate proposal per car across four cars, each with localised descriptions.
11	asset-metadata-relation	Add a car to an asset's relation field while preserving the entries already present.
12	asset-tags-visual	Vision - look at the image and tag it with the correct color, country, and quality.
13	research-car-price	Use the web-research tool to find a price and propose a written analysis (not internal guesses).
14	personalize-document	Create a target-group-scoped variant of an existing landing page for a given persona.
15	template-simple	Author and render a simple Twig template with a single editable headline.
16	template-advanced	Author a more complex template combining an image grid and a WYSIWYG editable.
17	create-news-object	Create a News object end-to-end - image, field-collection entries, and relations, in the correct folder.
18	landing-page-chevrolet	Build a complex landing page from web research: intro, FAQ, and an image carousel with linked content.

Metrics and grading

Every run records token accounting (input, output, cache-read, cache-write), server-side timing (model vs. tool time), tool-call counts, proposals, and emitted widgets. Automated checks (e.g. "this widget was emitted", "this proposal field equals X", "this tool was called", "no errors") run after the session; the operator's pass/fail verdict is the headline result.

Results

These results are spot-checks. Each figure is a single run; the agents and models are non-deterministic, so reruns will produce slightly different numbers and occasionally different pass/fail outcomes. Some failures are environmental rather than model quality - for example, Hugging Face inference endpoints that scale to zero can time out mid-run. Treat the table as directional, and re-run the eval tool against your own tasks before committing to a model for production.

One row per unique model, using that model's best (highest-scoring) run. Where a model was evaluated more than once, the run with the most passes is shown (ties broken by lower cost), so the date column varies per model. Non-cached input = total input − cache-read; it is the figure that best reflects effective input cost. Token counts are rounded to thousands; all runs are from June 2026.

Model	Date	Success	In	Out	Cached-in	Non-cached-in	Provider	Runtime	Estimated Costs
claude-sonnet-4.6	2026-06-23	18/18 (100%)	3,593k	61k	3,176k	417k	Copilot	25.5 min	3.12 $
claude-haiku-4.5	2026-06-09	17/18 (94%)	3,010k	41k	2,597k	412k	Copilot	10.2 min	0.88$
gpt-5.4-mini	2026-06-11	17/18 (94%)	3,468k	81k	3,010k	457k	Copilot	20.8 min	0.93$
qwen3p6-plus	2026-06-10	17/18 (94%)	5,431k	87k	4,485k	946k	Fireworks	40.0 min	1.19 $
gemini-3.5-flash	2026-06-11	17/18 (94%)	7,642k	67k	5,823k	1,820k	Copilot	29.2 min	4.2 $
kimi-k2p6	2026-06-10	16/18 (89%)	2,563k	56k	2,138k	425k	Fireworks	21.6 min	0.97 $
deepseek-v3.2	2026-06-22	16/18 (89%)	5,768k	63k	1,665k	4,103k	Huggingface	57.2 min	1.58$
Qwen3.6-35B-A3B	2026-06-19	14/18 (78%)	6,656k	131k	0	6,656k	Huggingface	30.2 min	1.12 $
minimax-m2.7	2026-06-22	14/18 (78%)	7,969k	108k	6,780k	1,190k	Huggingface	28.8 min	2.52 $
gemma4:31b-cloud	2026-06-11	13/18 (72%)	2,883k	19k	0	2,883k	Ollama	23.8 min	-
gemma-4-26B-A4B-it	2026-06-19	12/18 (67%)	10,111k	68k	0	10,111k	Huggingface	61.2 min	0.73 $
gpt-oss-120b	2026-06-09	10/18 (56%)	7,842k	87k	7,102k	740k	Cerebras	7.5 min	0.32 $
qwen3-32b	2026-06-19	10/18 (56%)	3,122k	137k	0	3,122k	Huggingface	59.7 min	0.28 $

claude-sonnet-4.6 is the only model to clear the whole suite (18/18) and is the accuracy leader, but it is several times more expensive per run than the efficiency leaders. Among the high-scorers, claude-haiku-4.5 is the most efficient - comparable non-cached input to Sonnet at roughly a third the cost and a quarter the runtime - which is why it remains the shipped default. The practical read: default to Haiku, and reach for Sonnet on the hardest document/creation agents where the extra reliability is worth the cost.

Per-model strengths, weaknesses, and failures

Each entry summarizes what the model is good and bad at, then lists the specific tasks its best run failed.

• claude-sonnet-4.6 (18/18): The only clean sweep - it passed every task, including the two hard document/creation tasks that trip most of the field (a full Chevrolet landing page with five product-teaser renderlets, and an end-to-end News object) plus the date and vision tasks. Strong caching (≈88%); the trade-off is cost - several times pricier per run than Haiku and ~2.5× slower.

  Failed: none.

• claude-haiku-4.5 (17/18): Best pass-rate-to-cost balance - one task off the top at the lowest cost and fastest runtime; reliable on object/asset updates, tagging, templates, listings, and document personalization. Its one real gap is Unix-timestamp/date arithmetic.

  Failed: update-event-object - set the wrong date and only converged after an explicit epoch was supplied.

• gpt-5.4-mini (17/18): Broadly correct across updates, research, templates, and the hard document tasks, with strong content quality; roughly twice as slow as Haiku.

  Failed: load-aston-martin-assets - found all assets but rendered a data-table instead of the required asset-preview widgets.

• qwen3p6-plus (17/18): Top-tier accuracy including dates and document tasks; the slowest of the 17/18 tier and prone to long single-inference stalls.

  Failed: template-advanced - no proposal; stalled in a 300 s inference timeout (environmental, not a logic error).

• gemini-3.5-flash (17/18): Very high pass rate across updates, listings, and templates, but high token and non-cached cost.

  Failed: landing-page-chevrolet - produced only an empty areablock skeleton and then thrashed on refine (serialization errors).

• kimi-k2p6 (16/18): Lowest total input of any model; fast and accurate on text/data tasks, but no reliable vision.

  Failed: load-red-cars (data-table instead of object-preview widgets); asset-tags-visual (blue car never tagged blue).

• deepseek-v3.2 (16/18): Strong content and research quality, but weak at date computation and expensive: caches poorly (≈29%) and is the slowest model.

  Failed: landing-page-chevrolet (car-teaser links unmet); update-event-object (wrong dates).

• Qwen3.6-35B-A3B (14/18): Gets listing content right but systematically mis-emits widgets and has no vision; no prompt cache inflates cost.

  Failed: load-red-cars, load-bmw-cars-table (wrong widget type), asset-tags-visual (vision hallucination), landing-page-chevrolet (empty-scaffold thrash).

• minimax-m2.7 (14/18): Reasonable on standard object updates, but skips required web research, is weak on vision and dates, and goes non-convergent on multi-part creates.

  Failed: research-car-price (no web research), update-event-object (dates), asset-tags-visual (vision miss), create-news-object (non-convergent, 96 tool calls).

• gemma4:31b-cloud (13/18): Cheapest output tokens and fine on simpler updates/listings, but stalls to empty answers on create/document tasks; no prompt cache.

  Failed: asset-tags-visual, update-event-object (dates), template-advanced (undefined CSS variable), and create-news-object / landing-page-chevrolet (no proposal - ended with an empty answer).

• gemma-4-26B-A4B-it (12/18): Passes simpler read/update tasks but falls into an "approve-minimal-first" non-convergence loop on anything multi-field; by far the most expensive (10.1 M input, 61 m).

  Failed: load-red-cars, update-car-object (only 5/7 fields), update-event-object, personalize-document, create-news-object, landing-page-chevrolet.

• gpt-oss-120b (10/18): Fastest wall-clock, but broadly unreliable for tool-driven work: leaks chain-of-thought as the answer, skips required tool calls, and hallucinates errors.

  Failed: eight tasks incl. smoke-list-classes (leaked reasoning, never called the tool), the data-query widgets, asset-tags-visual, research-car-price, and landing-page-chevrolet.

• qwen3-32b (10/18): Modest input footprint but the weakest tier: no vision, broken dates/relations (hallucinated IDs), no widget emission, and thrashes on templates/documents.

  Failed: eight tasks incl. the data-query widgets, asset-tags-visual, research-car-price, template-simple (42 failing calls), create-news-object, and landing-page-chevrolet.