Edit in GitHubLog an issue

Indexing

Indexing is how Adobe Commerce and Magento Open Source transform data such as products and categories, to improve the performance of your storefront. As data changes, the transformed data must be updated or reindexed. The application has a very sophisticated architecture that stores lots of merchant data (including catalog data, prices, users, and stores) in many database tables. To optimize storefront performance, the application accumulates data into special tables using indexers.

For example, if you change the price of an item from $4.99 to $3.99. the application must reindex the price change to display it on your storefront.

Without indexing, the application would have to calculate the price of every product on the fly, taking into account shopping cart price rules, bundle pricing, discounts, tier pricing, etc. Loading the price for a product would take a long time, possibly resulting in cart abandonment.

Indexing terminology#

Dictionary : Original data entered to the system. Dictionaries are organized in normal form to facilitate maintenance (updating the data).

Index : Representation of the original data for optimized reading and searching. Indexes can contain results of aggregations and various calculations. Index data can be always re-created from a dictionary using a certain algorithm.

Indexer : Object that creates an index.

Create custom indexers#

Adobe Commerce and Magento Open Source contain several indexers out of the box, but you might want to add your own if your customization requires data searches, which are not optimized by the default indexers.

This topic provides a high level description of how indexing is implemented from a developer's point of view, and practical advice for how to add your own indexer.

How the application implements indexing#

The following components are involved in the indexing process:

ComponentDescription
Magento_IndexerImplements:
  • indexer declaration
  • indexer running
  • indexer running mode configuration
  • indexer status
Magento\Framework\Mview
  • Allows tracking database changes for a certain entity (product, category, etc.) and running change handler.
  • Emulates the materialized view technology for MySQL using triggers and separate materialization process (provides executing PHP code instead of SQL queries, which allows materializing multiple queries).

Indexing types#

Each index can perform the following types of reindex operations:

  • Full reindex, which means rebuilding all the indexing-related database tables

    Full reindexing can be caused by a variety of things, including creating a new web store or new customer group.

    You can optionally fully reindex at any time using the command line.

  • Partial reindex, which means rebuilding the database tables only for the things that changed (like changing a single product attribute or price)

The type of reindex performed in each particular case depends on the type of changes made in the dictionary or in the system. This dependency is specific for each indexer.

The following figure shows the logic for partial reindexing.

Partial indexing workflow

Indexer status#

Depending on whether index data is up to date, an indexer status value is one of the following:

Database Status|Admin Status|Description valid|Ready|Data is synchronized, no reindex required invalid|Reindex Required|The original data was changed, the index should be updated working|Processing|Indexing is in progress

The database status can be seen when viewing the SQL table indexer_state. The admin status can be seen when viewing the indexer grid in Admin or when running the index status from the CLI.

The indexing mechanism uses the status value in reindex triggering process. You can check the status of an indexer in the Admin panel in System > Tools > Index Management or manually using the command line.

Using application lock mode for reindex processes#

Starting with 2.4.3, you can enable use_application_lock mode for reindexing through the use of environment variables or by configuring the app/etc/env.php file like so:

Copied to your clipboard
1<?php
2return [
3 'indexer' => [
4 'use_application_lock' => true
5 ]
6];

In case of a failure during the reindexing of a certain indexer, having this mode enabled will return a more accurate status of the indexer. The status can be obtained from the indexer grid in the Admin or through the bin/magento indexer:status command in the CLI.

An additional benefit of this mode is that the application will, internally, also see a more accurate status of the indexers and if an indexer failed, the application will now see this and the cronjob will pick up the indexer to try it again. Without this mode, it was necessary to manually reset the indexer when it failed. With this mode enabled, this should no longer be the case if the reindexing doesn't fail again during the next attempt.

Indexing modes#

Reindexing can be performed in two modes:

  • Update on Save - index tables are updated immediately after the dictionary data is changed.
  • Update by Schedule - index tables are updated by cron job according to the configured schedule.

To set these options:

  1. Log in to the Admin.
  2. Click System > Tools > Index Management.
  3. Select the checkbox next to each type of indexer to change.
  4. From the Actions list, click the indexing mode.
  5. Click Submit.

You can also reindex from the command line

The following figure shows an example of setting indexers to Update by Schedule:

Changing indexer modes

Mview#

The mview.xml file is used to track database changes for a certain entity.

For example part of Magento/Catalog/etc/mview.xml is tracking category to product relation described in the following record:

Copied to your clipboard
1<!-- ... -->
2<view id="catalog_category_product" class="Magento\Catalog\Model\Indexer\Category\Product" group="indexer">
3 <subscriptions>
4 <table name="catalog_category_entity" entity_column="entity_id" />
5 <table name="catalog_category_entity_int" entity_column="entity_id" />
6 </subscriptions>
7</view>
8<!-- ... -->

Explanation of nodes:

  • The view node defines an indexer. The id attribute is a name of the indexer table, the class attribute is indexer executor, the group attribute defines the indexer group.
  • The subscriptions node is a list of tables for tracking changes.
  • The table node defines the certain table to observe and track changes. The attribute name is a name of an observable table, the attribute entity_column is an identifier column of entity to be re-indexed. So, in case of catalog_category_product, whenever one or more categories is saved, updated or deleted in catalog_category_entity the execute method of Magento\Catalog\Model\Indexer\Category\Product will be called with argument ids containing ids of entities from column defined under entity_column attribute. If indexer type is set to "Update on Save" the method is called right away after the operation. If it set to "Update by Schedule" the mechanism creates a record in the change log table using MYSQL triggers.

A change log table is created according to the naming rule - INDEXER_TABLE_NAME + '_cl', in case of catalog_category_product it will be catalog_category_product_cl. The table contains the version_id auto-increment column and entity_id column that contains identifiers of entities to be re-indexed. For each table node the framework automatically creates MYSQL AFTER triggers for each possible event (INSERT, UPDATE, DELETE).

For the table catalog_category_entity triggers will be created with the following statements. INSERT operation:

Copied to your clipboard
1BEGIN
2 INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`);
3END

UPDATE operation:

Copied to your clipboard
1BEGIN
2 IF (NEW.`entity_id` <=> OLD.`entity_id`
3 OR NEW.`attribute_set_id` <=> OLD.`attribute_set_id`
4 OR NEW.`parent_id` <=> OLD.`parent_id`
5 OR NEW.`created_at` <=> OLD.`created_at`
6 OR NEW.`path` <=> OLD.`path`
7 OR NEW.`position` <=> OLD.`position`
8 OR NEW.`level` <=> OLD.`level`
9 OR NEW.`children_count` <=> OLD.`children_count`)
10 THEN INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`);
11 END IF;
12END

DELETE operation:

Copied to your clipboard
1BEGIN
2 INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (OLD.`entity_id`);
3END

The method Magento\Framework\Mview\ViewInterface::update is responsible for handling records in the changelog. The method is being called by CRON and it defines IDs to be re-indexed from the change log by last applied version_id and calls the execute method for each particular indexer with IDs as an argument.

How to reindex#

You can reindex by:

Indexers#

Magento Open Source implements the following indexers (use bin/magento indexer:info to list the indexers):

Indexer method nameIndexer classDescription
design_config_gridMagento\Theme\Model\Indexer\Design\Config
customer_gridMagento\Framework\Indexer\Action\EntityRebuilds the customer grid index. Not supported by the Update by Schedule indexing mode. See the Help Center article.
catalog_category_productMagento\Catalog\Model\Indexer\Category\ProductCreates category/products association
catalog_product_categoryMagento\Catalog\Model\Indexer\Product\CategoryCreates category/products association
catalog_product_priceMagento\Catalog\Model\Indexer\Product\PricePre-calculates product prices
catalog_product_attributeMagento\Catalog\Model\Indexer\Product\EavReorganizes the EAV product structure to flat structure
cataloginventory_stockMagento\CatalogInventory\Model\Indexer\Stock
catalogrule_ruleMagento\CatalogRule\Model\Indexer\Rule\RuleProductIndexer
catalogrule_productMagento\CatalogRule\Model\Indexer\Product\ProductRuleIndexer
catalogsearch_fulltextMagento\CatalogSearch\Model\Indexer\Fulltext

Adobe Commerce also provides the following indexers:

Indexer nameIndexer method nameIndexer classDescription
Product/Target Ruletargetrule_product_rule\Magento\TargetRule\Model\Indexer\TargetRule\Product\RuleIndexes product/rule association
Target Rule/Producttargetrule_rule_productMagento\TargetRule\Model\Indexer\TargetRule\Rule\ProductIndexes rule/product association
Sales Rulesalesrule_ruleMagento\AdvancedSalesRule\Model\Indexer\SalesRuleIndexes sales rule
Was this helpful?
  • Privacy
  • Terms of Use
  • Do not sell my personal information
  • AdChoices
Copyright © 2022 Adobe. All rights reserved.