Mobile Testing: The Balance Between Real Devices and Emulators

I believe that the debate of mobile testing on real devices vs. emulators is one of the oldest and most emotional debates in the mobile space over recent years.

In this blog, I would like to try and make “peace” between the parties who are in favor of each.

Before I give my POV on that, let’s clarify the exact meaning of “testing mobile apps” so we’re all on the same page (I wrote an entire book about it 😊).

Mobile app testing has a wide scope that includes unit tests, functional end-to-end (E2E) tests, performance and UX tests, security and accessibility tests, and compatibility tests. Some innovative apps will also require advanced audio and gesture test cases for chatbots, fingerprint and Face-ID authentication.

While we’re all familiar with the traditional test pyramid, when we apply it to mobile, the testing strategy often looks different – especially when we try to include the large scope and the market platforms scale into the testing cycle.


That’s why organizations are trying to balance between the test types, software development life cycle (SDLC) app phase, and other considerations to address the challenge of quality and velocity.

Emulators, on one hand, serve a very important goal during the SDLC. They greatly reduce costs to the developers and testers. Some would argue that they are faster to set up and execute. They have lower error rates and are already embedded in the developer’s environment in most cases, which is very convenient from a fast feedback perspective.

On the other hand, emulators are not running the formal carrier OS version – they do not run on the same real device hardware – they cannot cover all required environment-based testing like specific carrier configuration or unique sensors and gestures testing, and mostly – they cannot serve as a single Go/No-Go decision platform. End-users that are consuming the app from various locations and environments in the world are not using emulators but real devices with varying conditions, OS versions and many competing apps running in the background. Therefore, this should be the formal test bed for your advanced testing activity.

The best practice for mobile app testing should rely on a mix of tests that are handled by different personas in the mobile product team that are spread between emulators and real devices, based on the build phase. In the early sprint phases, when the features are only shaping up, it makes a lot of sense to run smoke tests, unit tests and other fast validations against emulators from the developer environment. Later in the build process, when the coverage requirements and the quality insights are greater, launching the full testing scope in parallel against real devices is the right way to go.

Real Devices vs. Emulators – LinkedIn Case Study

I presented last month at Android Summit 2017. The event was awesome, and I enjoyed great sessions and community networking. One of the presentations at this conference was given by Drew Hannay, Staff Software Engineer from LinkedIn.

LinkedIn (acquired by Microsoft) is now serving more than ½ a billion users across multiple platforms like Web and mobile. According to the company, it is constantly trying to attract new users, mostly of a young age (college graduates – the numbers of which are estimated at 180M by LinkedIn’s university campaign), or from expanded geographies such as India & China, which include 140M users in China alone.

LinkedIn used to suffer from poor quality, stability issues and bad reviews in the past. While trying to solve the quality issues, as well as keep growing, LinkedIn announced more than a year ago project Voyager: a new release model and SDLC strategy that is aimed to improve both quality as well as release velocity for the organization. LinkedIn shifted to an admirable 3×3 release strategy. As Drew presented, LinkedIn are now able to push a new version to production 3 times a day, every 3 hours from developer’s code commits – this is due to the announced project Voyager.



While the release cadence improved, allowing the LinkedIn product team to quickly adapt to changes, fix bugs faster and innovate, the end-user experience somehow dropped, and many of the mobile users are vocal about it in the app stores, stating that they would rather use the desktop browser instead of the mobile app. The thing about quality, especially in the mobile space, is that it is “a moment in time”. With such an aggressive release schedule to production, the build that was in production 3 hours ago is irrelevant from quality and stability perspectives, therefore, the product team must know at all times how the app works on real devices in production rather than assuming that it works just based on emulator testing.

As shown in the above image, LinkedIn are running as part of their test plan the entire test scope on 16 emulators in parallel. There is zero coverage on real devices, because as Drew stated – our implementation is agnostic across devices.

If we validate the above statement and strategy against the app store, and my own personal device experiences, the implementation isn’t that device-agnostic, and there are many escaped defects to production that are impacting real-device users.



The above experiences show: crashes of the app on Android devices when switching from Wi-Fi to real carrier networks, invites from mobile to various connections not working properly, sync issues between the LinkedIn feed and what’s shown on the browser version, installation issues on real devices and more.

When the majority of traffic to the LinkedIn app is coming from mobile devices, LinkedIn testing strategy should be tuned accordingly.


With the above stats and quality reality of LinkedIn users, the testing strategy needs to change. Future usage growth is expected to come from India, China and non-U. S demographics, building the case for different devices, OS, form factors and network conditions.


LinkedIn needs to base their mobile testing strategy on real life personas operating from different geographic locations, varying conditions, background apps and more.

Testing on emulators is essential and should be kept as part of the strategy, but it cannot be the only platform for testing this app – as seen above, it does not guarantee continuous quality and UX.


Perfecto Blog

The best mobile games to play while your family watches Thanksgiving football

Thanksgiving is a time to reconnect with loved ones, eat lots of food, and all of that jazz, but once the festivities start to wind down, folks tend to head to the couch to watch whatever football is happening for Turkey Day.

If you find this particular Thanksgiving ritual to be something of a snoozer, or if you prefer to multitask with your post-feast entertainment, we have a few games that are perfect to play so you can look like you’re participating in the family bonding while secretly tuning out. These games are addictive, but light enough that you can pop into conversation and play without sound.

[Read more]
148Apps » iPhone, iPad, Apple Watch and iPod touch App Reviews and News (RSS Feed) | 148Apps

Both of Nintendo’s mobile game payment models are fine

Super Mario Run

In the past decade, smartphones have rapidly developed into these incredibly innovative devices that are more powerful than one might have ever imagined for a gadget that fits into the palm of your hand. Obviously, all that power has to be used for something; it doesn’t take much to make a phone call or send a text. Multitasking between the web, productivity tools, social media, and media streaming are some of the most common ways to use a smartphone to its full potential, but surprisingly (or maybe unsurprisingly) mobile gaming has also taken a strong foothold among users as well.

Although many smartphones have specs that can rival and even outperform traditional computers, the way mobile games are developed and played are held to different standards than that of PCs or consoles. Oftentimes, mobile games are free – to start, at least – but mobile games can sometimes require in-app purchases to progress past a certain point. While this isn’t always a problem, sometimes it can be. In a recent segment on The Tonight Show Starring Jimmy Fallon, entertainer Jack Black revealed that his son fell victim to these in-app purchases in a game targeted towards children, racking up a massive $ 3,000 bill on in-game currency.

Speaking from personal experience, companies like Google, Apple, and Amazon have done a much better job cracking down on these “unauthorized” in-app purchases over the years. It isn’t hard to receive refunds. However, it’s important to note that not all charges are made by unknowing children. Many times, these in-app purchases are made by adults who are free to spend their money as they choose, sometimes getting addicted and willingly spend thousands in the process. It’s not hard to see why games like this keep popping up despite their exploitative nature.

When Nintendo announced last year that they would start making mobile games, there was a question of which model the company would go with. Nintendo has employed the use of microtransactions in some of their more modern console games like Super Smash Bros. and Mario Kart 8, keeping some characters and DLC behind an additional paywall, albeit not game-breaking. It was possible for Nintendo to head down that route if they wanted. As it turned out, they didn’t at first.

For their first mobile title, Super Mario Run, the process was pretty simple: You can download the game for free and play the first three levels, but if you want to progress through the story beyond that, you’ll need to pay $ 10 to unlock everything. While some might consider $ 10 a steep price – especially for a mobile game – I think it’s an acceptable asking price for a solid mobile-optimized Mario game. Additionally, the game has already gone on sale for $ 5 at one point, and there was an update included to add more content just a year after its launch at no extra cost.

Super Mario Run

But Nintendo received a lot of flak for asking $ 10 for the game (and probably more for inexplicably requiring an Internet connection to play, despite the game being mostly a singer-player experience). Since Super Mario Run, Nintendo has released two more mobile games: Fire Emblem Heroes and, as of yesterday, Animal Crossing: Pocket Camp. These games deviate from the “freemium” model of Super Mario Run and instead are considered free-to-play with optional microtransactions.

Fire Emblem Heroes players can pay between $ 1.99 and $ 74.99 at a time to purchase orbs, the “hard currency” of the game, and users can buy them as often as they like. However, as one player found out after spending $ 1,000 in the game, even spending an exorbitant amount of money won’t guarantee you a complete set of heroes when randomly generated outcomes are involved. Fortunately, there are ways of earning orbs for free just by playing, and many people find that the game is perfectly enjoyable without spending a dime.

Animal Crossing: Pocket Camp operates similarly, albeit the microtransactions don’t reach as high for a single purchase. Users can spend between $ 0.99 and $ 39.99 at a time to buy Leaf Tickets, the Animal Crossing equivalent to Fire Emblem’s Orbs. Obviously, I haven’t played Animal Crossing: Pocket Camp for more than a day, but there appears to be no shortage of opportunities to earn Leaf Tickets in the game. Judging by Fire Emblem Heroes, I expect people will be able to have a great time on the game without spending any money if they don’t want to.

That said, I’m torn between Nintendo’s two models. Given the choice of paying a reasonable set price or being able to play for free with the option of paying for microtransactions, I prefer the set price. But that’s adult me with a job talking. Children, on the other hand, don’t always have the option of using real money for games, so being free-to-play with optional microtransactions appears to be the best scenario to give everybody a chance to enjoy the game regardless of wealth status. Interestingly enough, free-to-play with in-app purchases is also the model that makes the most money, even when you pit one successful Nintendo franchise against another.

I’m not the biggest fan of microtransactions in general, but as long as those microtransactions don’t involve game-breaking benefits (where the game is heavily skewed in favor of those with real cash to spend, thus becoming less “free-to-play” and more “pay-to-win”), then I find that they’re easy to ignore. Overall, I agree with Nintendo’s stance that a one-time purchase a la Super Mario Run is more consumer-friendly, but their free-to-play models are equally as enjoyable in my opinion, and I haven’t paid anything towards either one (and am admittedly having a really good time with Animal Crossing: Pocket Camp). I think Nintendo has done a good job with their mobile game lineup so far and look forward to checking out titles they release in the future.

Readers, what are your thoughts on Nintendo’s two mobile game models? Do you think that a one-time purchase is preferable to free-to-play with optional in-app purchases, or the other way around? Let us know your thoughts in the comments below! – Latest videos, reviews, articles, news and posts

iOS11 and Android Oreo (8.0) are Shocking the Mobile Market: Key Findings in Perfecto Factors #8 Magazine

Quarterly, I release a digital market report, that highlights trends, usage of devices and desktop browsers as well as predictions for the following quarter.

What I’ve found over the past 2 quarters, is that the market is becoming more fragmented than ever, creating huge implication on mobile application developers and testers.

Important Market Events

During last quarter, Apple and Google released 2 major operating systems on to the market – iOS11 and Android Oreo (8.0).

Compared to previous releases like iOS10 and iOS9, Apple is seeing a dramatic reduction in the adoption of the new platform as well as iPhone8/iPhone 8+ purchases. In addition, iOS11 is suffering from significant quality-related issues around battery drain, performance and functionality that has already driven Apple to release an immediate patch release, and 2 other patches (11.0.2, 11.0.3), with another beta version maturing toward a 2nd patch release (11.1).

apple iOS 11.0.1

On Android, Google has released Android Oreo to Pixel and Nexus devices – though most vendors are still lagging. This release is also exhibiting significant defects around networking, battery drain and other functional-related issues.

Android Oreo appears to be disabling mobile data on some pixel and Nexus devices, fix in the works

With these 2 latest OS versions slowly ramping up and requiring a lot of R&D attention from the mobile application vendors, the previous releases of both iOS and Android demand continuous support since most users are still using them.

andriod Oreo 8.0

Application Development Implications

If we come to understand the key implications on the mobile market for the recent trends, we can see the following:

  1. iOS & Android markets are increasingly fragmenting – the iOS platform is now divided into 3 major iOS versions (9.3.5, 10.3.3, iOS11.x). Testing and development need to accommodate this complexity and ensure that existing test code can still be executed across these 3 platforms and that new features implemented are well supported at least on the latest 2 iOS families. Similarly, Android Oreo, as the future platform for major brands like Samsung and other key OEMs, needs to be well tested already by application vendors to identify gaps in advance. To do so, teams must include either a Pixel or a Nexus device in their test lab from an Android perspective.
  2. Test automation maintenance is expected to grow – the above point of 3 OS versions for iOS, as well as Android platform fragmentation, necessitates a structured strategy that can meet the application testing requirements, e.g. iPhone X that will be launched in November 2017, introduces FaceID recognition instead of today’s Fingerprint authentication that is used by iPhone 8 and below devices. Automation code will have to be adjusted to support the specific device/OS/feature complexities.
  3. Test execution intelligence – as a common practice, teams ought to be on top of their test suites data, from a test suite sizing perspective, execution length, test duplicates and other data management aspects. Especially with the growing trend of fragmentation and complexity in automating more, teams should have a way of injecting intelligence into their suites either throughout executions or post executions, and drive data-driven decisions that can help optimize the overall testing activities throughout their pipelines.

With the above challenges in mind, now’s the right time to provide a planning tool that can guide application teams for mobile and web, on what to include in their current and future cycles, how to optimize their testing strategy and be prepared for Q4 and beyond.

To read more on the newest report findings, and access the latest 14 different Indexes covering EMEA, North America, and APAC, please refer to this link.

Perfecto Factors

Perfecto Blog

The best mobile threat defense is mobile threat detection

As enterprises push ahead with mobile-first strategies – and employee smartphones and tablets increasingly become business tools – the importance of mobile threat defense (MTD) is growing.

Using mobile threat detection and defense, however, is no small task; the technology must cover applications, networks and device-level threats to iOS and Android phones and tablets to be effective.

“We talk about mobile threat defense, rather than detection – the reason being these solutions not only detect, but also can prevent and remediate threats,” said Dionisio Zumerle, research director for mobile security at Gartner.

The MTD market is growing in terms of adoption, and has started to attract attention from endpoint protection platform (EPP) vendors and in other related markets, according to a recent report from Gartner.

To read this article in full, please click here

Computerworld Mobile