Edit in GitHubLog an issue

MFTF test versioning and backward compatibility

Goals and requirements#

  1. Release MFTF tests as a separate package on repo.magento.com.
  2. Define the versioning strategy for MFTF test packages.
  3. Outline what is considered a backward incompatible change to MFTF tests.
  4. List of what should be implemented.

Backward compatibility for MFTF tests#

When a test undergoes changes, but achieves the same testing results as before and remains compatible with potential test customizations, this is defined as a 'backwards compatible' change.

Types of changes:

  • Test Flow change (Test/ActionGroup) - A backwards compatible modification of a test flow would not diminish the original set of actions in the test. Some changes may change an action's sequence (behavior), but they allow any extension to achieve the same test results without changing the test extension (e.g a 'merge file').
  • Test Entity change (Data/Section/Page/Metadata) - Compatible modifications of entities are 1) adding new entities or 2) updating a value of an existing entity in a way where the test will NOT require updates.
  • Test Annotation change - Annotations can be changed without limitation and will always be considered a backward compatible change, but removing or changing a <group /> annotation will be considered a backward incompatible change.
  • Changes which delete and/or rename a (Test/Action Group/Data/Metadata/Page/Section/Action)'s id attribute will be considered a backward incompatible change. Changing a reference to a data entity will also be considered a backward incompatible change.

Versioning policy#

The approach of defining what each release should include was taken from Semantic Versioning.

3-component version numbers:

Copied to your clipboard
1 X.Y.Z
2 | | |
3 | | +-- Backward Compatible changes (bug fixes)
4 | +---- Backward Compatible changes (new features)
5 +------ Backward Incompatible changes

Z release#

Patch version Z MUST be incremented if only backward compatible changes to tests are introduced. For instance: a fix which aims to resolve test flakiness. This can be done by updating an unreliable selector, adding a wait to an element, or updating a data entity value.

Y release#

Minor version Y MUST be incremented if a new, backwards compatible test or test entity is introduced. It MUST be incremented if any test or test entity is marked as deprecated. It MAY include patch level changes. Patch version MUST be reset to 0 when the minor version is incremented.

X release#

Major version X MUST be incremented if any backwards incompatible changes are introduced to a test or test entity. It MAY include minor and patch level changes. Patch and minor version MUST be reset to 0 when the major version is incremented.

Implementation tasks#

  1. Add Semantic Version analyzer to be able automatically define the release type of the MFTF tests package.
  2. Update publication infrastructure to exclude tests from magento2-module package type.
  3. Introduce publication functionality for publishing magento2-test-module package type.
  4. Create a metapackage with test packages specifically for Adobe Commerce and Magento Open Source.

Version increase matrix#

Entity TypeChangeVersion Increase
ActionGroup<actionGroup> addedMINOR
<actionGroup> removedMAJOR
<actionGroup> <action> addedMINOR
<actionGroup> <action> removedMAJOR
<actionGroup> <action> type changedPATCH
<actionGroup> <action> attribute changedPATCH
<actionGroup> <argument> with defaultValueaddedMINOR
<actionGroup> <argument> without defaultValue addedMAJOR
<actionGroup> <argument> removedMAJOR
<actionGroup> <argument> changedMAJOR
<actionGroup> <remove> <action> addedMAJOR
<actionGroup> <remove> <action> removedMAJOR
Data<entity> addedMINOR
<entity> removedMAJOR
<entity> <array> addedMINOR
<entity> <array> removedMAJOR
<entity> <array> <item> removedPATCH
<entity> <data> addedMINOR
<entity> <data> removedMAJOR
<entity> <required-entity> addedMAJOR
<entity> <required-entity> removedMAJOR
<entity> <var> addedMAJOR
<entity> <var> removedMAJOR
<entity> type addedMINOR
<entity> type removedMAJOR
<entity> type changedMAJOR
Metadata<operation> addedMINOR
<operation> removedMAJOR
<operation> changedMINOR
<operation> child element addedMINOR
<operation> child element removedMAJOR
Page<page> addedMINOR
<page> removedMAJOR
<page> <section> addedMINOR
<page> <section> removedMAJOR
Section<section> addedMINOR
<section> removedMAJOR
<section> <element> addedMINOR
<section> <element> removedMAJOR
<section> <element> selector changedPATCH
<section> <element> type changedPATCH
<section> <element> parameterized changedMAJOR
Test<test> addedMINOR
<test> removedMAJOR
<test> <action> addedMINOR
<test> <action> removedMAJOR
<test> <action> changedPATCH
<test> <action> sequence changedMAJOR
<test> <action> type (click, fillField, etc) changedPATCH
<test> <actionGroup> ref changedMINOR
<test> <before/after> <action> addedMINOR
<test> <before/after> <action> removedMAJOR
<test> <before/after> <action> changedPATCH
<test> <before/after> <action> sequence changedMAJOR
<test> <before/after> <action> type (click, fillField, etc) changedPATCH
<test> <before/after> <actionGroup> ref changedMINOR
<test> <annotations> <annotation> addedPATCH
<test> <annotations> <annotation> changedPATCH
<test> <annotations> <annotation> GROUP removedMAJOR
<test> <remove> <action> addedMAJOR
<test> <remove> <action> removedMAJOR
Suite<suite> addedMINOR
<suite> removedMAJOR
<suite> <include/exclude> <group/test/module> addedPATCH
<suite> <include/exclude> <group/test/module> removedPATCH
<suite> <before/after> <action> addedMINOR
<suite> <before/after> <action> removedMAJOR
<suite> <before/after> <remove> <action> addedMAJOR
<suite> <before/after> <remove> <action> removedMAJOR
<suite> <before/after> <action> changedPATCH
<suite> <before/after> <action> sequence changedMAJOR
<suite> <before/after> <action> type (click, fillField, etc) changedPATCH
<suite> <before/after> <actionGroup> ref changedMINOR

⃰ - <action> refers to any of the available MFTF Actions.

Was this helpful?
  • Privacy
  • Terms of Use
  • Do not sell my personal information
  • AdChoices
Copyright © 2022 Adobe. All rights reserved.