Duplicate Detection (Experimental)

caution

This is an experimental feature, built on the experimental Embeddings feature. Configuration, data format and APIs may still change in breaking ways between releases.

Catalogues accumulate the same record twice: the same car imported from two feeds, the same photo uploaded under two filenames. Duplicate detection finds those pairs — including the ones that are not byte-identical — and gives you a screen to review and resolve them.

It reuses the vectors and content hashes the Embeddings feature already produces, so detection costs no extra inference calls.

The three things to understand first

A group, not a pair. Detection picks one element — the reference — and collects everything that looks like it. One group is one reference plus its candidate members, each with a similarity score. The screenshot above shows the group anchored on …/W111/sedan/black with five members scoring 76 % down to 71 %.

Two tiers. An exact duplicate has an identical content hash. A fuzzy duplicate merely looks alike, and a score says how much. Both end up in the same review screen, see How Detection Works.

Finding is not resolving. Detection only ever proposes. Nothing is changed in your data until you act on a group, and the two things you can do are decline it (these are not duplicates) or merge it (they are — fold one into the other), see Review Workflow.

The workflow at a glance

1. Configure which classes and which asset tree to look at, which elements to exclude, and how strict to be, see Configuration.
2. Run detection, from the configuration screen, the review screen, the CLI, or a cron, see Runs and Progress.
3. Review each group: the reference, its members, and their scores.
4. Decline what is not a duplicate — it moves to the group's Declined tab and stops coming back.
5. Merge what is: pick the two elements, carry values across, re-point anything that referenced the duplicate, move its children, and decide what happens to the leftover.

Features

• Objects and images. Data objects are compared on their field values; images on what the picture looks like. Documents and text assets are never deduplicated.
• Guided merging. A wizard walks through field merge, incoming references, child elements and the fate of the duplicate, skipping the steps that do not apply.
• Nothing happens behind your back. Detection writes findings, never content.
• Two thresholds. One controls what is worth storing, the other what is worth showing — so you can quieten the review queue without re-running anything.
• Structural sense. A folder is never a duplicate of its contents, and product variants under one parent are suppressed by default instead of flooding the queue.

Requirements

• Embeddings enabled, with an object_dedup usage for data objects and an image_dedup usage plus image in the asset scope for images.
• The bptb_duplicates_permission user permission for the review screens, see User Permissions. The configuration screen is admin-only.
• A worker consuming pimcore_backend_power_tools_duplicates_queue.
• The Object Merger bundle, for merging data objects. Without it the review screens still work, and Compare & merge reports that it is missing.

Pages in this chapter

• Review Workflow: the review screens, declining, and every step of the merge wizard.
• How Detection Works: exact and fuzzy tiers, how scores are computed, what is stored versus shown, and the filters that drop matches outright.
• Configuration: every setting, in the editor and in YAML.
• Runs and Progress: incremental versus full runs, following one, and cancelling it.