Application Testing Guide introduction

The Commerce framework actively leverages various testing strategies to ensure product and code quality. Product quality matters to the end users working with the storefront, admin panel, web APIs and CLI. Product quality tests verify that the user gets expected results while interacting with the system.

Code quality is important to developers working with the codebase including:

Code quality matters for:

Automated tests are required by definition of done for any code changes.

Product Quality Tests

Code Quality Tests

Functional

Functional tests are mainly used for system tests at a very high level by remote controlling a browser. The application is treated as a black box, and tests happen from a user perspective.

The Functional Testing Framework (MFTF) is the main way to do functional testing. It uses XML files to run browser testing.

API Functional

The Web API testing framework enables you to test the Web API from the client application point of view.

For more information, see the Web API functional testing.

Integration

Integration tests run PHP code in varying degrees of isolation. They tend to be a lot more low-level then functional tests. Because they do not utilize a browser to execute the tests, they can be a lot more granular in what they test. They also tend to run a lot quicker then functional tests.

For more information, see Running Integration Tests.

JavaScript

Much of the functionality is provided with the help of sophisticated JavaScript. JavaScript tests ensure the frontend portion of functions as expected.

For more information, please see the Extension Developer Guide on JavaScript Tests.

Static

Static code analysis checks that PHP code follows the Magento 2 coding standards and best practices. They are executed during continuous integration using the bin/magento tool.

See the magento dev:tests:run documentation for more information, using the test type static.

PHPStan

PHPStan is a static analysis tool for PHP. The application integrates PHPStan by default. It is available for developers to use in their own work, located at Magento\Test\Php\LiveCodeTest::testPhpStan().

It is run on changed *.php files using rule strictness level 1 and detects issues such as:

Suppressing errors

To ignore errors in a particular piece of code, use the phpstan:ignore annotation.

// phpstan:ignore "Error message"

The error message text is optional. It is possible to use the * wildcard within messages.

/**
 * Class ClassWithIgnoreAnnotation
 */
class ClassWithIgnoreAnnotation
{
    /**
     * Test method.
     * phpstan:ignore "Method level error"
     */
    public function getProductList()
    {
        // phpstan:ignore "Method Magento\PhpStan\Formatters\Fixtures\ClassWithIgnoreAnnotation::testMethod() invoked with 2 parameters, 1 required."
        $this->testMethod('test1', 'test2');
        // phpstan:ignore "Method * invoked with 2 parameters, 1 required."
        $this->testMethod('test1', 'test2');
        // phpstan:ignore
        $this->testMethod('test1', 'test2');
        $this->testMethod('test1', 'test2'); // phpstan:ignore
    }
    /**
     * @param string $arg1
     * @return string
     */
    private function testMethod(string $arg1)
    {
        return $arg1;
    }
}

Adding files and directories to PHPStan blacklist

If some files are broken on purpose, add them to the blacklist to ignore them.

The blacklist is located at dev/tests/static/testsuite/Magento/Test/Php/_files/phpstan/blacklist/.

lib/internal/Magento/Framework/Interception/Test/Unit/Config/ConfigTest.php
lib/internal/Magento/Framework/Cache/Backend/Eaccelerator.php

Exclude error messages from the analysis results

If some error messages are not relevant for the current testing scenario, use regular expressions to filter them.

The PHPStan configuration file is dev/tests/static/testsuite/Magento/Test/Php/_files/phpstan/phpstan.neon.

Example:

ignoreErrors:
    # Ignore PHPStan\Rules\Classes\UnusedConstructorParametersRule
   - '#Constructor of class [a-zA-Z0-9\\_]+ has an unused parameter#'

Exclude files and directories from analysis

As needed in some cases, you can exclude specific directories and files using the excludes_analyse section in the PHPStan configuration file: dev/tests/static/testsuite/Magento/Test/Php/_files/phpstan/phpstan.neon.

Example:

excludes_analyse:
   - %rootDir%/../../../*/_files/*
   - %rootDir%/../../../dev/tests/*/Fixtures/*
   - %rootDir%/../../../dev/tests/*/tmp/*
   - %rootDir%/../../../pub/*

Unit

Unit tests are used to check a single unit of PHP code in isolation. They are usually written during development using test-driven development (TDD).

Because they do not require the full application stack to be initialized, they run an order of magnitude faster then integration tests.

For more information, see Running Unit Tests.

Please refer to the article Writing testable code for more information on what to keep in mind when starting with TDD.

The bin/magento tool provides a common entry point to execute any of the tests, which can be useful for continuous integration. Please see the System Administrators Guide on Running Tests for more information.

Where to find the tests in the file system

Each test type described above corresponds to a subdirectory in <repo_root>/dev/tests, as shown here:

<repo_root>
    <dev/tests/>
        acceptance      (since v2.2.4)
        api-functional
        functional
        integration
        js/jasmine
        setup-integration
        static
        unit

Each of these test types must satisfy different requirements before the MFTF can execute them.

MFTF tests are kept within its respective module folder:

<extension_repo_root>
├── <Module1>
│   ├── ...
│   ├── Test
│   │   ├── Unit
│   │   ├── Integration
│   │   └── Mftf
│   │       ├── TestSuite
│   │       └── composer.json
│   └── ...
├── <Module2>
├── <Module1SampleData>
└── <Module2SampleData>