A while back, I, along with my colleague Geera, presented a webinar – ‘Multi-device Testing: Challenges, Considerations and Solutions’. As a preparation for this webinar and, in general, being a part of the Quality Assurance team, we tried some tools and simulators to test multi-device eLearning to study to what extent they could help. Below is a quick summary of our experiences for some of the key ones.
Please note: These tools and simulators have been explored with a specific focus on their capability to test multi-device eLearning, and not websites or web applications.
1. Synchronous Testing Tools
Synchronised testing tools help in performing the same interaction automatically across a number of devices and browsers at the same time.
a. GhostLab
GhostLab is a tool that allows testers to wirelessly pair multiple iOS, Android, Windows, and BlackBerry devices to their computer and carry out synchronous testing. GhostLab involves concurrent testing on actual physical devices and configurations, and you can check real user experiences. Changes (e.g. changing a page or displaying content on clicking a button) can be simultaneously viewed on all connected devices.
Advantages
- Allows us to check real user experiences
- Results of change can be seen simultaneously
- Visuals and alignment can be checked
- Joint reviews can be conducted
- Pre-release verification can be conducted
- Allows you to immediately check code changes
- Can be helpful for testing text-based courses
Below is an image of multi-device testing in progress using GhostLab.
b. Adobe Edge Inspect
Adobe Edge Inspect is another tool that displays synchronised output on physical devices. It allows testing on iOS and Android mobile devices, as well as Kindle Fire. With Adobe Edge, you need to install the main application on a desktop computer, and launch the course through a Chrome browser. You can then access the synchronized course through the Edge Inspect app on connected devices.
Advantages
- Launches courses through the Chrome browser on the main desktop and through the Edge Inspect app on connected devices
- Loads HTML pages
- Allows synchronous control of course-level navigation and page-level interactivities for courses structured to have separate URLs for pages/content shown on click
Below is a snapshot of multi-device testing in progress using Adobe Edge Inspect.
Synchronous Testing tools do provide a multi-device testing interface, but it’s difficult to simultaneously keep one’s eyes and ears on so many devices!
Limitations of Synchronous Testing tools for eLearning
- Difficult to simultaneously keep our eyes on so many devices!
- Difficult – or pretty much impossible – to check audio synching on multiple devices simultaneously!
- May need to do course-level navigation and page-level interactivities manually on connected devices
- May not be able to check browser-related issues depending on how courses launch
2. Simulators
Simulators create a virtual environment or object which can be a physical system or an object. Simulators come with both the hardware and the software which runs on it to create the real experience.
a. BrowserStack
BrowserStack is a cloud-based simulator that we found to be useful for checking text alignment on a specific device-OS-browser combination and catching visual issues.
In case of cloud-based simulators, content loading, as well as speed and smoothness of animations, can vary based on the connection, which may lead to inaccurate issues being logged. Remote simulators can also cause some difficulties when it comes to testing interactivities, as option can’t be selected unless content is fully and properly loaded on the remote machine.
b. SDK Simulators
SDK simulators, which are local simulators, can also be helpful for testing some parameters. With the iOS and Windows simulators, we could check content, alignment, and layouts quite reliably. However, we did encounter some inaccuracies with the Android emulator.
After trying some simulators for testing eLearning, we realised we cannot ignore the fact that actual learner experience cannot be completely checked, since physical devices are not being used – so there can be inaccuracies and performance mismatches based on the configuration and resolution of the machine running the simulator.
Limitations of Simulators for testing eLearning
- Cannot test actual learner experience
- Cannot check real-time physical device aspects, i.e. battery life and charging, and actual bandwidth speed
- Cannot check content/functionality dependent on physical device features
- Cannot check gestural navigation (custom interactivities)
- Inaccuracies and performance mismatches based on simulator machine configuration and resolution
While tools and simulators help in testing, they just aren’t quite the real thing. As mentioned in an earlier blog post – The Proof Of The Pudding Is In Multi-device ‘Physical’ Testing, there is no alternative to testing on physical (real) devices. Physical ‘real’ devices are an absolute must for you to experience your eLearning exactly as your users will.
