Thieves, refrigerators and witchcrafts – How wearables can challenge you in testing.
Marilyn Monroe in a famous interview said: “What do I wear in bed? Why, Chanel No. 5, of course.” I go to bed wearing a wearable, which wakes me up when there is the best gap in my REM phase. Is this the world of future? Yes. Welcome.
With Marilyn: An Evening/1961Photography by Douglas Kirkland
Not to start from scratch, let’s just say that we can wear in technology from head to toe (literally!) From the smart socks (Sensoria, Owlet, Netflix), bra (OmBra), up to smart suit with NFC technology from Samsung. Smart bands, fitness trackers and smart coaches, devices to track our activity, they need no introduction. Undeniably, best known wearables are smartwatches, also in the business world recognizable as’ Key wearable product type „. Why? They give us ability to connect to knowledge within finger tap and quick access to data with both hands still free. Or maybe, because the size of our smartphones is still growing …?
Each one of wearable should be tested based on its top use cases. But where to start from? …
Challenge No. 1: Sources
The first collision: the lack of easily accessible information on how smartwatches work. Achievable information is scattered, incomplete and often out of date due to the extremely dynamic development of wearables themselves, but also the development’s tools. There is no substitute for direct interaction with smart watch and already available applications.
Challenge No. 2: Environment
Environment completely determines the approach for testing. Depending on the platform, applications can have different architecture. They may have Host/companion app (Android) on your phone, be an extension to a mobile application (iOS) or work as a standalone app available only for smartwatch (Tizen). They can be native, web or hybrid, and it is related to different API availability, and finally the application’s behavior (installation, updates, it’s states, the behavior of the foreground and background). The base is to know your application from scratch! You must abandon your comfort zone and go out to meet developers. After all, from your understanding depends future app quality.
http://www.samsung.com/global/galaxy/gear-s3/
In case of mobile applications, OS fragmentation was important, because of app compatibility. For wearables, paired smartphones OS is important, as their features depends on it. Smart watch’s life is too short, the cost of maintaining support for previous versions is unjustified (good example: the war for support for Android Wear 2). In 2016, Tizen IDE survived 2-fold update within a few months, iOS released a new Watch OS 3 and just after that Watch OS 3.2 (currently still in beta tests, while new Watch OS4 already announced!) and we are still waiting for Android Wear devices to be updated to version 2 … All changes associated with significant changes in the functioning of smart watches, not just bug fixing. And all during one project. Sounds like challenge? In between updates, discrepancies might be formed. The application stops responding, because developer used a method that does not exist on earlier OS of paired device, not mentioning bugs typical for UI.
But Smart Watch themselves are not free from shortcomings: unexpected restarts, disconnecting from the smartphone, and even unpairing. Simply browse through forum posts on Google.
To make sure that what you see is an issue in your app, repeat, repeat and repeat tests … Yes, wearables testing is time-consuming process. This must be considered and justified in the eyes of the customer. But this challenge is on management site…
Challenge No. 3: Connections
Smart Watches do not seem to be so smart after all, as their full functionality depends on paired smartphone (Android from ver.4.3, iOS ver. 8.2.) The BLE protocol is the same for all platforms (open source), but its performance depends not only on the version of the OS, but also from the smartphone manufacturer.
„Supported smartphones and features a small vary depending on your region, service provider, and device manufacturer.” It is reasonable to limit the number of devices we need to support.
Testing application on wearables, we put the most attention to its behavior when the Bluetooth connection is interrupted, lost or disabled and recovered again. Synchronization of data in each case is crucial. But the nature of Bluetooth connectivity is still not fully stable, protocol is still being improved. Don’t be surprised when one day your connection to smartwatch will be ‘stolen’. This 2-step process can be easily disturbed by trying to connect to our device by others, before the right bond is set properly. This stealing connection effect is visible mainly in a test environment where many smartphones were already paired with your wearable, and took their place on their own list of trusted devices (white lists). Without proper Bond, wearables become victims of the race for the connection. First come first served. You end up with wearable connected but not paired, without the possibility to connect again until reset.
It is worth noticing, that apps on smartwatches connect to their Hosts / companion apps without inducing their full activity or UI. iOS treats the principle of „keep it simple” very seriously, so that we can act on smartwatch, although not being registered yet on iPhone.
Challenge No4: Usability and Human Experience Testing
Main expectations for wearable applications are that they should be „easy to use”, intuitive, simple and fun. Test weight changes. Usability gains priority. Users want to use their favorite applications in a way that it reflects the interactions on their smartphone. It is not enough to reduce the mobile application and push it onto that small screen, it needs to be adjusted (Morphing applications, more: http://miter.mit.edu/critical-features-for-multi-device-applications ). It also includes storing data in the cloud, synchronizing application state across devices and the ability of applications to pick up from where the user left off on a different device. Where is challenge? Well, all the data can be changed freely, on multiple devices.
The flow of an app must be natural. All interactions should be easy, designed in a way to meet user’s 'paths of habits’ and platform requirements. Variety of gestures and interaction grows, they are not always obvious, often require additional support from the code. Any problems related to buttons and gestures are bound to lead to trouble as User Experience is disturbed. Accessibility and adaptability to small screens include: adjusting fonts (style, size), the marquee effect (auto scrolling long texts), zoom, etc. These adjustments can disrupt navigation in application and apparently affect its performance. 'In Wild testing’ gets real importance here. It’s not enough anymore ‘click through’ the application in the comfort of your office, but test it in the “real world”. We shift to Human Experience testing, which involves emotional, physical and sensory reactions, social expectations and interactions. ‘ The closer the device becomes to the human, the more important 'Human’ testing becomes” (more: 'Wearables: Testing the Human experience” by Gerie Owen (see https://www.slideshare.net/JosiahRenaudin/wearables-testing-the-human-experience?next_slideshow=1)) There is just no replacement for the real use of wearable in a crowded bus at rush hour, when a storm rages outside…
Challenge no.5: Notifications
Notifications deserve special attention. Platform dependent architecture, types, logic behind their interactions is complex and strongly linked to other regions of application. Let’s have a look at iOS. Notifications are displayed here only for apps listed in Watch manager. What will happen if we pair watch from another platform (Tizen, Android)? Cross-ptaform pairing is supported, but what notifications will I receive and where? Is there any security risk involved? Well, everything depends on the implementation …
Let’s add some more devices in IoT communication. Imagine, you change the parameters of your coffee mug from smartwatch. The matrix of test cases gets really complicated. The battle is about proper displaying notifications on wearable paired with smartphone, and correct interactions (answering, rejecting, etc.) As always, the key is not to spam user. The check for tester skills is creating an adequate test matrix, which in these cases is extremely rich … More? Let’s pair another smartwatch with different settings for notifications. Remember that each platform supports pairing for multiple devices and switching should be done automatically. Hmmm … so where is my notification?
Challenge No 6: Performance
Smartwatches have not enough memory, inefficient processors, and low batteries. It is crucial to test application in terms of resource consumption. The lack of clear expectations and guidelines for application performance on wearables, makes this topic problematic and more intuitive than documented. Does it mean that the problem does not exist?
The weakest point is battery life. To extend operating time of wearables, there are many features implemented (Wi-Fi disconnection, screen off, reducing brightness, gray scale, etc.), however, they do not refer directly to battery draining issue in paired device. Constant communication, data exchange costs. Again, add to this matrix additional IoT device. Test matrix based on their battery states (empty, full, charging, etc.) gets challenging. Note, that for IoT devices, their battery critical level often involves a change in reduction of communication and often related UI changes in companion app.
Challenge No7: Tools
Wearables are not always available (expensive, short project, not yet for commercial sale). We often test prototypes or have to use emulators/simulators, with their own shortcomings. While emulators (available for all platforms) are quite well suited for functional and/or regression tests, so far, they are not able to fully cover cases with syncing/connectivity issues, capturing real data and providing the real-time feedback you expect from a wearable. Creativity is then extremely desirable. It is not enough to sit in your favorite chair and click through app. You must reveal your skills of electronic, physics, a little of chemist and sometimes even a cook. We need to incorporate the roles of distracted users, who wearable for heating coffee hide in the refrigerator. How the sensors will manage then?
Usefulness of logs from smartwatches is not so obvious. Logs are available mainly through the IDE or cmd (dlog, logcat). However, their quality is rather poor, especially considering crash log. Identification of the problem and its repair is often not possible without a complete reproducibility. Additional logs must be written by developers, if they are to be useful.
The easiest way to register an issue is to make a screenshot. But here you can encounter challenges, where Android Wear is an example. Screenshots are available here only from the Manager. They are received on paired phone as notification and each file has the same name. All screenshots must be forwarded, shared with changed name or data will be irretrievably lost. Currently, Smart watches also do not have appropriate tools which would allow them to correctly compress data. Easy screen record seems not supported then on any platform, not without your third recording device.
Challenge No 8: Future
Believing all the available reports, predictions, business fairies, number of smartwatch’s users will increase till 2020 approximately 4 times. The trust in technology grows. Smartwatch fans are big advocates of the thought that smartphones will be replaced in the future by wearables (43%, more: http://www.gemalto.com/review/Pages/the-future-of-wearables-video.aspx). Smaller and more dedicated sensors are already available and achievable, components that should make smartwatches more independent from the technology of smartphones. We will interact with them by „witchcrafts” (gestures in the air) and transmit data through a handshake (more: http://www.fiercewireless.com/tech/at-t-refines-bone-conduction-technology-for-gestures-mobile-devices; https://www.macrumors.com/2017/02/09/apple-patent-data-sharing-gestures-wearables ).
Testing wearables and their apps is as exciting as demanding. In this rapidly evolving market, testers are facing many traps and challenges: variety of environments, platform differences, tools availability, additional interactions with user and IoT, etc. The significance of tests changes. Usability gains „priority” as the tangible, aesthetic pleasure of products paves the way for our emotional connections to wearables.We can love Smartwatches or hate them, but nothing can change the fact, that they took their place in our hearts and wrists.
Autor: Magdalena Lisowska