IBM Engineering Lifecycle Management platform consists of several domain specific solutions, Engineering Test Management (ETM) being one of these solutions. Out of its capabilities, Envionment Management in ETM might considered to be an undervalued feature.

It can be considered to be as an undeniable fact that the importance of test management is growing along with the complexity of the projects. Teams not only should be concerned about the Requirement – Test coverage in a one dimensional traceability space, but should also be concerned about the status of the verification of the requirements as well as the second dimension to the traceability. Which might be simply depicted in the figure below:

Where, the trace between requirement and test case presents a very basic traceability from requirement and test case coverage point of view. However, including the test results to this concept will enhance the information we can get out of traceability, because it is not enough now to have a test case, it is also important for the requirement to pass the tests.

But this still, cannot be the whole story. Because in a typical system development scenario, it should not be enough that the requirement has passed the verification scenario. Has it passed the verification scenario in some mandatory conditions? Especially, the conditions that we cannot control within the system but the system still interacts with. Hence, has the system been verified according to its intended use environment? In other words, what about the third dimension?

Accoding to this figure, what could be the verdict of the requirement’s verification? Wouldn’t it be misleading if the latest result was collected in Summer? Is there a risk that the previous failed cases will be shadowed by the latest successful result?

In this article, we would like to highlight a very rich, yet undermined feature that IBM ETM proposes to manage the verification of requirements for different conditions.

A Test environment refers to the setup where testing activities take place. For testing of software, it typically includes the hardware, software, network configurations, and any necessary system dependencies required to run tests. However if a system is considered instead of being pure software, the definition of “environment” would be quite extended. It can be generalized to be the outside conditions that the system is in interaction with or affected by. For instance for an atomspheric combustion engine, the air pressure (hence the altitude) is definitely a test envrionment. We should want to test and record the performance of the engine on different altitudes.

Test Environments are also called as Lab Resources in ETM, because generally the system under test is being tested in a simulated environment to mimic the conditions of the intended use environment, hence a test lab.
Before delving into the topic, we would like to split the usage of the test environments into two parts, namely the basic and the advanced scope of using test environments. These two scopes can be listed as follows:

Basic usage of test environments,

• Defining Test Environments
• Using test environments in test plans
• Associating test envrionments with test cases
• Running test cases against test environments
• Reporting on test results on test environments

More advanced usage of test environments:

• Creating test lab resources out of test environments
• Registering test lab resources in a lab resource reservation system
• Requesting a lab resource for a scheduled test
• Fulfilling requests
• Creating test cells out of test lab resources

In this article we will be focusing only on the basic scope, which can still promise significant value even without exploring the advanced features. On a prospecting article next, we also would like to delve into the advanced usage as well.

A new environment type can be defined by clicking on the green cross:

Make sure the “Set as test environment type” is selected. Otherwise you cannot use this type in specifiying test environments. Then define the allowed values as an enumarated list of the new environment type:

Once the environments and their allowed list of values are defined, next step is to use these values to scope the test plans with them. This will help us to define enviornmental boundaries to test plans

To start scoping the test plan, go to the Test Environments of the test plan:

This figure indicates there is only one instance of testing for the performance of the engine. Hoever, we may expect different test results for different conditions. Hence, there should be multiple instances of this test case to verify the performance requirements on different conditions.

To address this issue, we multiply the test cases with the test environments and generate an instance of the same test case for every environment. We call these instances as “Test Case Execution Records”

Based on this simple scenario, there are 3 different reports that can be generated as follows:

1. Requirement Test Case Coverage: Are all requirements have a traceability link to test cases
2. Requirement Test Result Coverage: Are all requirements on all environments verified? Are there any failed tests on specific environments.
3. Environment Verification: Are all Requirements on a specific environment successfully verified? - This is also an important metric if a quick release is targeted for a specific environment. There might be some issues on other environments but we don’t have to wait to release for the environments that are verified.

The usage of test envrionments would bring a great value to the verification and validation processes. Some of which could be listed as follows:

1. Consistency and Reproducibility: Managing test environments ensures that tests are executed in the same conditions every time, making results reproducible. This consistency is critical when debugging issues or comparing results across different test cycles.
2. Efficient Resource Management: Proper test environment management in ETM helps optimize resources. Teams can define environments based on the needs of various test phases, ensuring that the right environment is available at the right time.
3. Support for Complex Testing Scenarios: System often depend on a variety of components and / or Subsystems, such as ECU, motor, hydrolic elements, mechatronical elements etc. A well-defined test environment enables testing of complex scenarios, including system integration and end-to-end testing.
4. Traceability: ETM’s ability to link test environments to specific tests, requirements, and defects enhances traceability. It provides clear documentation on which environment was used to execute which tests, helping teams maintain a transparent testing process.
5. Collaboration Across Teams: Testing environments are often used by multiple teams (development, QA, operations). ETM helps coordinate between these teams to ensure that all necessary environments are available and properly configured for testing.

In conclusion, managing test environments is a critical component of effective test management, and IBM Engineering Test Management (ETM) offers a robust solution for organizing, allocating, and tracking test environments throughout the development lifecycle. Properly configured test environments ensure that tests are run consistently, efficiently, and accurately, leading to reliable and actionable results. By integrating test environment management into your workflow and following best practices, you can significantly improve the quality of your products.