Customer Duplicates Service

CMF's Customer Duplicates Service helps you find, merge, and avoid duplicate entries. It consists of three parts:

• Part 1: mechanisms for searching duplicates of a given customer
• Part 2: duplicates index
• Part 3: duplicates view

Parts 1 and 2 configure background processing. The duplicates view is where a user sees, validates, merges, or declines potential customer duplicates in Pimcore Studio.

Part 1: Searching Duplicates of a Given Customer

This works directly against the database via the customer object list. The duplicate service finds active customers whose configured fields match the given customer, using simple lowercase and trim transformations so comparisons are case-insensitive.

Define the field combinations to match in the CMF configuration file, under customer_duplicates_services.duplicateCheckFields. You can configure multiple field combinations; for example, check firstname/lastname/street/zip/city first, and if no duplicates are found, check firstname/street/zip/city/birthDate.

Performance tip

Add customer save handlers that trim the duplicate check fields, then list those fields under customer_duplicates_services.duplicateCheckTrimmedFields. This removes the need for a trim operation in the resulting query, which can be a significant performance boost.

Samples for Part 1

<?php 

// Create a new Customer Instance
$customer = new Customer();
$customer->setBirthDate(new Date('1982-12-07'));
$customer->setFirstname("Markus");
$customer->setLastname("Moser");
$customer->setZip("5020");
$customer->setPublished(true);
$customer->setActive(true);

// Get an object list with duplicates for the new customer instance (set limit to 1)
$service = $container->get('CustomerManagementFrameworkBundle\CustomerDuplicatesService\CustomerDuplicatesServiceInterface');
$duplicates = $service->getDuplicatesOfCustomer($customer, 1);


// If duplicates exist and "checkForDuplicates" is activated in the CMF config file, an exception will be thrown when
// trying to save the new customer and a duplicate exists.
try {
    $customer->save();
} catch(DuplicateCustomerException $e) {
    print "save failed - duplicate found: " . $e->getDuplicateCustomer() . PHP_EOL;
}

//get duplicates of an existing customer
$existingCustomer = Customer::getById(12345);
$duplicates = $service->getDuplicatesOfCustomer($existingCustomer, 1);

Part 2: Duplicates Index

The duplicates index searches globally for fuzzy-matching duplicates. Found duplicates appear in the customer duplicates view (Part 3), where a user can merge them manually.

To keep the duplicate search performant, the data is stored in a dedicated format in the duplicate index. By default, this uses several MariaDB tables, but you can implement DuplicateIndexInterface to build a DuplicateIndex for Elasticsearch instead.

Configure the duplicates index in the configuration file. Data and logic for storing duplicates in the index is configured as follows.

Example Config

pimcore_customer_management_framework:
    customer_duplicates_services:
        duplicates_index:
            enableDuplicatesIndex: true

            duplicateCheckFields:
              - firstname:
                  soundex: true
                  metaphone: true
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\SimilarText

                zip:
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\Zip

                street:
                  soundex: true
                  metaphone: true
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\SimilarText

                birthDate:
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\BirthDate

              - lastname:
                  soundex: true
                  metaphone: true
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\SimilarText

                firstname:
                  soundex: true
                  metaphone: true
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\SimilarText

                zip:
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\Zip

                city:
                  soundex: true
                  metaphone: true
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\SimilarText

                street:
                  soundex: true
                  metaphone: true
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\SimilarText


              - email:
                  metaphone: true
                  similarity: \CustomerManagementFrameworkBundle\DataSimilarityMatcher\SimilarText
                  similarityThreshold: 90

            dataTransformers:
              street: \CustomerManagementFrameworkBundle\DataTransformer\DuplicateIndex\Street
              firstname: \CustomerManagementFrameworkBundle\DataTransformer\DuplicateIndex\Simplify
              city: \CustomerManagementFrameworkBundle\DataTransformer\DuplicateIndex\Simplify
              lastname: \CustomerManagementFrameworkBundle\DataTransformer\DuplicateIndex\Simplify
              birthDate: \CustomerManagementFrameworkBundle\DataTransformer\DuplicateIndex\Date

Define the field combinations to match under customer_duplicates_services.duplicates_index.duplicateCheckFields.

For each field in these combinations, configure how it is indexed with four options:

• soundex: set to true to make the field relevant for soundex matching. Enable this for text fields where soundex matching makes sense, for example firstname. It is not useful for a field like zip.
• metaphone: the same idea as soundex, using a different phonetic algorithm. You can combine soundex and metaphone, though enabling both is sometimes unnecessary and wastes resources. Enable both if you are unsure.
• similarity: soundex and metaphone matching produce many false positives, both because some fields (like zip) should not use them at all, and because the algorithms themselves are imprecise. Configure a SimilarityMatcher (see DataSimilarityMatcherInterface) to filter these out. Every potential duplicate found via soundex or metaphone is compared using the configured SimilarityMatchers, and only counts as a real duplicate if all fields pass; otherwise it is a false positive.
• similarityThreshold: each SimilarityMatcher has a default threshold. Optionally override it with a custom threshold passed to the SimilarityMatcher.

Console Command

Use this console command to calculate and update the duplicates index. Configure it as a cron job if needed.

bin/console cmf:duplicates-index

Options:

• -c: calculate potential duplicates. Run this as a cron job, for example once a day.
• -a: analyze false positives. Logs them, by default in the plugin_cmf_duplicates_false_positives table.
• -r: recreate the index. Rebuilds the full index for all customers.

Part 3: Duplicates View

The duplicates view is where a user manually checks, merges, or declines potential duplicates found via the duplicates index. It gets populated when the duplicates index console commands run, and its usage is largely self-explanatory.