MFTF test versioning and backward compatibility

Goals and requirements

1. Release MFTF tests as a separate package on repo.magento.com.
2. Define the versioning strategy for MFTF test packages.
3. Outline what is considered a backward incompatible change to MFTF tests.
4. List of what should be implemented.

Backward compatibility for MFTF tests

When a test undergoes changes, but achieves the same testing results as before and remains compatible with potential test customizations, this is defined as a 'backwards compatible' change.

Types of changes:

• Test Flow change (Test/ActionGroup) - A backwards compatible modification of a test flow would not diminish the original set of actions in the test. Some changes may change an action's sequence (behavior), but they allow any extension to achieve the same test results without changing the test extension (e.g a 'merge file').
• Test Entity change (Data/Section/Page/Metadata) - Compatible modifications of entities are 1) adding new entities or 2) updating a value of an existing entity in a way where the test will NOT require updates.
• Test Annotation change - Annotations can be changed without limitation and will always be considered a backward compatible change, but removing or changing a <group /> annotation will be considered a backward incompatible change.
• Changes which delete and/or rename a (Test/Action Group/Data/Metadata/Page/Section/Action)'s id attribute will be considered a backward incompatible change. Changing a reference to a data entity will also be considered a backward incompatible change.

Versioning policy

The approach of defining what each release should include was taken from Semantic Versioning.

3-component version numbers:

Copy
   X.Y.Z
   | | |
   | | +-- Backward Compatible changes (bug fixes)
   | +---- Backward Compatible changes (new features)
   +------ Backward Incompatible changes
Copied to your clipboard
   X.Y.Z
   | | |
   | | +-- Backward Compatible changes (bug fixes)
   | +---- Backward Compatible changes (new features)
   +------ Backward Incompatible changes

Z release

Patch version Z MUST be incremented if only backward compatible changes to tests are introduced. For instance: a fix which aims to resolve test flakiness. This can be done by updating an unreliable selector, adding a wait to an element, or updating a data entity value.

Y release

Minor version Y MUST be incremented if a new, backwards compatible test or test entity is introduced. It MUST be incremented if any test or test entity is marked as deprecated. It MAY include patch level changes. Patch version MUST be reset to 0 when the minor version is incremented.

X release

Major version X MUST be incremented if any backwards incompatible changes are introduced to a test or test entity. It MAY include minor and patch level changes. Patch and minor version MUST be reset to 0 when the major version is incremented.

Implementation tasks

1. Add Semantic Version analyzer to be able automatically define the release type of the MFTF tests package.
2. Update publication infrastructure to exclude tests from magento2-module package type.
3. Introduce publication functionality for publishing magento2-test-module package type.
4. Create a metapackage with test packages specifically for Adobe Commerce and Magento Open Source.

Version increase matrix

⃰ - <action> refers to any of the available MFTF Actions.

Last updated 11/22/2023

Was this helpful?

Yes

No