The two most powerful warriors are patience and time.
—Leo Tolstoy
If we are talking about mobile testing approaches, we need to dedicate a post on how to source mobile testing. This is an issue for every test organization, and it seems there is no general solution for everyone.
In order to structure the sourcing of mobile testing, I will classify the solution into two main approaches, as many other professionals do. The first one will be “in-the-lab” testing and second one will be “in-the-wild” testing. As the names suggest, the main classification of sourcing is based on the location of the testers. As we have discussed and (hopefully) agreed on in previous sections, some part of the mobile testing should be done while on the move. This becomes quite feasible with an in-the-wild approach.
Of course, you could keep yourself moving even though you are using in-the-lab sourcing, but the chances are a bit lower. The following indicates the pros and cons of each of my classifications. Benefits are marked with a plus sign and tradeoffs with a minus sign.
In-the-Lab Testing
- In-house testing is done by testers who are located on company premises and who are under the company payroll.
o Easy to control and manage [+]
o Staff has extensive domain/business know-how [+]
o Hard to find, hire, and fire testers [–]
o Owning physical devices is expensive [–]
o Testers may not represent real users [–]
- Insourced testing is done by testers who are located on company premises but are under the payroll of the test service provider (third-party outsourcer).
o Easy to control and manage [+]
o Easy to ramp up/ramp down the capacity [+]
o Physical devices can be supplied by the contractor [+]
o Staff can lack domain/business know-how [–]
o A complete and high-quality test basis is needed [–]
- Outsourced testing is done by testers who are located off-site and who are under the payroll of the test service provider (third-party outsourcer).
o Easy to ramp up/ramp down the capacity [+]
o Physical devices can be supplied by the contractor [+]
o Staff can lack domain/business know-how [–]
o Difficult to control and manage due to location/time zone differences [–]
o A complete and high-quality test basis is needed [–]
In-the-Wild Testing
- Beta testing is done by the internal employees (generally are not selected from testers) of the company or by the real users of the mobile app found and organized by the company.
o Fewer physical devices are purchased by the company (may use BYOD, that is, bring your own device) [+]
o Easy to ramp up/ramp down the capacity [+]
o High variety of physical devices [+]
o Carrier and network features are included in tests [+]
o Testing is on the move [+]
o Testers can lack domain/business know-how [–]
o Testers can lack mobile testing know-how [–]
o May damage company reputation since testers can be real users [–]
o May have security, reliability problems [–]
o A complete and high-quality test basis is needed [–]
- Crowd-sourced (community) testing is done by real users who are controlled, organized, and given incentives by the test service provider (e.g., Mob4Hire, Applause, UserTesting, etc.)
o No physical device is purchased by the company (using BYOD) [+]
o Easy to ramp up/ramp down the capacity [+]
o High variety of physical devices [+]
o Carrier and network features are included in tests [+]
o Testing is on the move [+]
o Hard to control and manage [–]
o Users can lack domain/business know-how [–]
o Users can lack mobile testing know-how [–]
o May have security, reliability problems [–]
o May damage company reputation since testers are real users [–]
o A complete and high-quality test basis is needed [–]
Which one is better? The answer to this question depends a lot on the software development life-cycle model you are using. If you have an agile approach with small teams and frequent releases, you will definitely have fewer testers on board and you need to build a lean organization. If you are following a waterfall approach within a more crowded organization where more complex products are built with many more features and requirements and less frequent releases, you may construct a hierarchal test team with distinct roles and responsibilities.
In any case, if you have enough budget and staff capacity, I suggest that you build your in-house staff first. In order to manage and control the test activities and to own the main responsibility of mobile testing, presence of an in-house test staff is critical.
Of course, you can completely outsource your test activities; however, the bottom line is that an in-house company representative should be in charge of delivering a high-quality mobile app to the users. If you own and centralize at least the governing and controlling side of your mobile testing activities, you will definitely have better mobile apps.
In order to supplement your in-house testing and add an independent perspective to it, you may choose the option of outsourcing. Aforementioned outsourcing approaches will bring you many benefits, such as:
- Staff flexibility
- No HR-related costs of hiring and firing testers
- SLA-based contracts
- More real devices and less real device ownership
- Know-how transfer from a professional testing service provider
In the current demanding and complex era of the mobile app industry, blending traditional in-house testing approaches with outsourcing solutions seems to make your testing better. Keeping a small, effective, and talented in-house staff and delivering the major part of your mobile testing to a professional vendor is one of the wisest solutions you may end up with.
All in all, people nowadays are talking about the hot “lean” phenomenon, and testing organizations can stay lean if and only if they make use of proper outsourcing mechanisms.