Combining Model-Based and Risk-Based Testing for Effective Test Case Generation

COMBINING MODEL-BASED AND RISK-BASED TESTING FOR EFFECTIVE TEST CASE GENERATION

Due to the increasing size and complexity of software systems, it becomes infeasible to perform exhaustive tests with limited resources. For instance, TV platforms are in fast transformation from old electromechanical systems to complicated software systems. Recently introduced Smart TVs combine web browsing, on demand streaming and home networking with traditional/standard functions of TV sets. According to the statistics provided by the YouGov market research agency, the number of Smart TV users will be around 2 billion. These users tend to be less tolerant against software failures that degrade the TV functionality. Therefore, it is important to create test cases that can detect these failures as soon as possible.

Model-based testing (MBT) is used for automatically generating test cases based on system models. The contents of these models influence the effectiveness of the generated test cases. In these models, for all components of system under test, ‘states, inputs, outputs and probabilities for the transitions’ should be determined. State represents the current status of the TV (e.g. turned off, turned on, input selection, etc.) A transition is used to change the current state to next state within a specific probability (e.g. from turned off state to turned on state).

Figure 1 – States and Transitions

Risk-based testing is a type of effective software testing that prioritizes the tests of features and modules in software, based on the risk of failure.

Model-Based Testing (MBT) systematizes test case generation based on models that represent the desired behavior of the system under test. In the following iterative approach, it is explained how to refine system models based on the principles of risk-based testing.

First, a system model with Markov Chains is defined, in which transitions among system states are annotated with probability values. These values are usually assigned based on functional importance or usage profile.

In Figure 2, the approach is depicted. In our approach, they are initially equal to each other. The model is provided to a MBT tool. The tool traverses the model based on transition probabilities and generates a set of test cases (2). These test cases are executed (3), while the memory usage profile is collected (4). The collected profile reveals test paths that lead to memory leaks. The transition probabilities are updated (5) and reflected to the system model (6) so that these test paths are used more often during test case generation in the next iteration.

Figure 2 – Approach

We performed a case study to apply the approach by updating system models which is in Figure 3 according to the detected memory leaks in Figure 4. We updated the transition probabilities in model according to the test result, so that the features with high memory leak are visited more.

Figure 3 – Recorded Memory Usages

Figure 4 – The Top Level Test Model

We were able to detect 3 critical failures with our approach. Although the number of test cases and the testing time are reduced.

Reference: Gebizli, C.S., Metin, D., & Sozer, H. (2015). Combining model-based and risk-based testing for effective test case generation. In Proceedings of the 9th workshop on testing: Academic and industrial conference—practice and research techniques (pp. 1–4) (ICST Companion)