MFTF Commerce-supplied tests
At this stage, the MFTF Commerce-supplied check is not required to pass technical review. The pass/fail status of this check is ignored. We are still observing the behavior of this set of tests and will start requiring these tests at a later time, with sufficient notice.
The Functional Testing Framework (MFTF) is a browser-based acceptance testing framework used to validate the functionality of a Adobe Commerce or Magento Open Source site. Running Commerce-supplied MFTF tests aids Extension Quality Program (EQP) QA efforts in assessing critical functionality of an instance with the app or extension installed.
Commerce supplied MFTF tests are not required to pass for technical review. The outcome of this testing is used to assist with QA efforts and provide feedback on functional quality.
For information on how to setup, create and modify MFTF tests, see Introduction to the Functional Testing Framework.
What testing is for
Commerce-supplied MFTF tests validate critical user flows via browser-based testing against an operational Commerce instance.
MFTF runs tests in a browser, using Selenium and Codeception, to emulate user behavior. MFTF tests are designed to be extensible and can be used in conjunction with other Commerce-developed MFTF code coverage or your own vendor-supplied tests. The results of this testing will inform QA about the state of the extension or app and allow them to prioritize manual testing.
When testing is done
Commerce-supplied MFTF tests will be run for all Marketplace extension and app submissions on Commerce 2.4 release lines. The tests will also be run for extensions running Magento Open Source 2.4.
The MFTF tests executed are:
- StorefrontGuestCheckoutTest
- AdminCreateCustomerWithCountryUSATest
- AdminCreateNewCustomerOnStorefrontTest
- StorefrontCustomerCheckoutTest
- AddConfigurableProductToOrderFromShoppingCartTest
- StorefrontReorderAsGuestTest
- StorefrontUpdateWishlistTest
- StorefrontClearAllCompareProductsTest
- MoveConfigurableProductsInComparedOnOrderPageTest
- CreateInvoiceWithZeroSubtotalCheckoutTest
- CreateInvoiceWithShipmentAndCheckInvoicedOrderTest
- AdminCreateCreditMemoPartialRefundTest
- AdminCreateOrderWithSimpleProductTest
- AdminCreateSimpleProductTest
- AdminConfigurableProductCreateTest
- AdminCreateCategoryTest
- AdminCreateAndEditBundleProductSettingsTest
- AdminCreateAndEditConfigurableProductSettingsTest
- AdminCreateAndEditVirtualGiftCardProductSettingsTest
What is being checked
MFTF tests verify the proper functioning of basic Commerce workflows. They are meant to mimic the same code paths that the end user would take, by automating the clicks and data entry that the user would do. The codebase contains thousands of MFTF tests.
Tools and environments used
The test infrastructure executes Commerce-supplied MFTF tests in the most up-to-date version of Commerce in the 2.4.x and 2.3.x release lines, as well as the most up-to-date software compatible with that release. The test infrastructure follows the recommended setup for an installation, MFTF setup, and system configuration. You can use Cloud Docker to create a similar environment.
Reading the error report
MFTF returns two types of results, displayed as MFTF Magento
within the developer portal:
- Simplified results showing the status of each test executed
- Allure XML results
See MFTF Reporting for further information.
- Test failures are generally due to:
- The MFTF test flow being breaking due to changes to Commerce user flows as intended by the extension or app under test. These can be fixed by using vendor-supplied MFTF Tests.
- A failure in the Commerce user flow due to an unintended side-effect of the extension or app under test. This may be a type of regression bug and will be tested and verified by manual QA as part of EQP technical review.
- The Allure results returned to Marketplace can be downloaded and displayed as an Allure report for simple consumption and identification of failure points.
Troubleshooting
- Ensure that your tests generate, execute and pass in the specified environment configuration
- Tests must be in the
Test/Mftf
directory within the extension - MFTF tests within this directory must follow the standard directory structure, separating ActionGroups, Tests, Pages, Sections, etc into their own directories
- Ensure that tests names do not clash with existing tests
- If you are extending from, or merging into, an existing test (or relying upon its entities), it must be required as a Composer prerequisite
- The MFTF tests will be part of the final package that is made available to your customers. Do not include any sensitive or confidential data in test comments or code
- If necessary, include a README, or other instructions, within the
Test
directory, to explain any setup steps or caveats for running your MFTF tests - Ensure that any necessary credentials or user authorization (for example, to communicate via API key to your backend) is explained and supported via credential management in tests
- Always follow the MFTF Best Practices and use the MFTF Tips & Tricks.