DESIGNING A LONGEVITY TEST FOR A SMART TV
Smart TVs have an important place in consumer electronics market with new technologies that they introduce to users. Quality of a product is an important criterion in consumer electronics market. End users become less tolerant to software based problems on their TV systems. This situation forces the manufacturers to produce more reliable products. To increase reliability we need to verify the products by detailed testing with more test cycles.
Detailed testing of such a complex system needs a long period of time. Test approaches that adopt the automated testing methods provide cost reduction, improving the product quality and effective time and resource usage. However, that is only the execution part and execution tools cannot find defects without effective test design. Quality and effectiveness of executed test plan completely depends on test design. For that reason, we improved test process by integrating Model Based Testing (Test Design) and Test Automation (Test Execution).
When you buy a product, for example, you are informed that this product’s life time is 10 years. This information is given by analyzing the materials, components, the hardware of the product. What if the software of a TV is analyzed and a lifetime is given until a major problem occurs?
During automated tests, so-called “torture tests” are performed. Torture tests aim at discovering how the system behaves under sustained use. One of the torture tests is Longevity test. The goal of Longevity Test is to discover how the system behaves under sustained use. This test helps us to ensure that the television is fully functional after some long period of sustained activity and also helps us to predict how long this system may continue before it functionally breaks. We are trying to predict software lifespan of Smart TVs.
– Smart TVs are sent to 50 field testers.
Hereby, these testers are actually the employees of the company as well; however, they are selected from various departments in the company. The selection process starts with an announcement made to all the employees in the company, informing them about the product, and data to be collected. The selection is performed to maximize the diversity among the testers in terms of the following properties: i) age, ii) department, iii) peripheral devices being used at home, and iv) type/availability of network connection at home.
– Modules being visited more by real users are detected by analyzing those log files.
– Then a test model is defined with Markov Chains, in which transitions among system states are annotated with probability values. A usage model is a formal statistical representation of all possible uses of the system. A model structure is a directed graph in which nodes are states of use and arcs are possible transitions. A model-based test (MBT) technique is utilized for the test design and systemizes test case and test script generation based on models that represent the desired behavior of the SUT.
– The SUT model is provided as an input to an MBT tool, which automatically generates a set of test cases and test scripts by traversing the possible behavioral scenarios on the system model.
– Coverage of each generated test case is 100%, which means each generated test case visits all possible modules detected by usage log analysis. So, those test cases simulate daily usage. They visit all possible modules by going through different possible paths. If we generate 365 different test cases, we are very close to simulate 1 year of a real user.
– Execution of each generated test script takes approximately 1,5 hours automatically.
– By executing those 365 test cases takes approximately 23 days and at the end of this 23day-test, we simulate approximately 1 year usage of the system under test.