Data Architecture and Indexing Process
Depending on the Product Index implementation, there are two different Product Index data architectures and ways for indexing: Simple Mysql Architecture and Optimized Architecture.
For indexing itself the provided Pimcore console commands can be used.
Simple Mysql Architecture
- Pimcore object data is transferred directly to the Product Index.
- After every update of a Pimcore object, the changes are directly written into the Product Index.
- Only used for
\Pimcore\Bundle\EcommerceFrameworkBundle\IndexService\Config\DefaultMysql
Updates of dependent objects (like child objects, variants that inherit data, related objects) are not transferred into the index automatically.
Console Commands for Simple Mysql Architecture
- For manually updating the whole index use following command:
php bin/console ecommerce:indexservice:bootstrap --update-index
- If you need to create or update the index structures you can use:
php bin/console ecommerce:indexservice:bootstrap --create-or-update-index-structure
- By default, Pimcore assumes that the product class is
Pimcore\Model\DataObject\Product
. If you have a different product class name, please use the--object-list-class
param and provide the listing class name that should be used.
For further details (e.g. only updating certain product types, apply list conditions) see
--help
section of theecommerce:indexservice:bootstrap
command.
Optimized Architecture
-
In the optimized architecture, object data is transferred not directly to the Product Index.
-
In this case a so called store table is between the Pimcore objects and the Product Index. This store table enables to ...
- ... update the Product Index only if index relevant data has changed. Therefore the load on the index itself is reduced and unnecessary write operations are prevented.
- ... update the Product Index asynchronously and therefore update also dependent elements (children, variants, ...) of an updated Pimcore object without impact on save performance.
- ... rebuilding the whole Product Index out of the store table much faster since no direct interaction with Pimcore objects is needed.
-
After every update of a Pimcore object, the changes are written into the store table and all child objects of the updated object are added to the so called preparation queue (see later). As a consequence a periodic full update should not be necessary any more.
-
Used for optimized mysql, elasticsearch, ...
Console Commands for Optimized Architecture
For updating data in index following commands are available.
- For process the preparation queue and update Pimcore objects to the index store table, use following command. This command should be executed periodically (e.g. all 10 minutes).
php bin/console ecommerce:indexservice:process-preparation-queue
- To update the Product Index based on changes stored in the store table use the following command. This command should be executed periodically (e.g. all 10 minutes).
php bin/console ecommerce:indexservice:process-update-queue
- For manually update all Pimcore objects in the index store use following command. As stated before, this should only be
necessary for an initial fill-up of the index. After that, at least Product Index Store and Pimcore objects should always
be in sync. It is important to execute
ecommerce:indexservice:process-preparation-queue
andecommerce:indexservice:process-update-queue
periodically though.
php bin/console ecommerce:indexservice:bootstrap --update-index
By default, Pimcore assumes that the product class is
Pimcore\Model\DataObject\Product
. If you have a different product class name, please use the--object-list-class
param and provide the listing class name that should be used.
- Invalidate either the preparation queue or the index-update queue. This is usually only needed during development when the store table is out of sync. Reset the preparation queue for instance when your product model returns updated data for a field.
php bin/console ecommerce:indexservice:reset-queue preparation --tenant=MyTenant
php bin/console ecommerce:indexservice:reset-queue update-index --tenant=MyTenant
- If you need to create or update the index structures you can still use:
php bin/console ecommerce:indexservice:bootstrap --create-or-update-index-structure
For further details see
--help
section of the commands.
Paralelization of Indexing
All indexing commands include the parallelization trait of webmozart.
Thus indexing can be parallelized very easily by adding the --processes=X
option to the command.
Be aware that depending on the command too many parallel processes might cause deadlocks on database.