Major Challenges of IoT Testing in Detail

According to Gartner, the number of IoT devices in use is expected to grow to more than 20 million by 2026. This exponential growth is important to consider when we think about the complexity and nature of the testing of IoT devices – which range from machines that automatically place orders to the supermarket to self-driving cars – because this is one of the biggest challenges device integrators and manufacturers are facing and it will only get tougher from here.

Testing needs to be effective. But how? Let’s look at some of the considerations for IoT testing devices, and tips that can be very helpful.

Tester’s Concern: Endless IoT Platforms

The hardware and software for each IoT are different with their own set of specifications, pros, and cons. Application software will also integrate with IoT devices, issue commands to the device, and analyze data gathered by the device.

Because there are numerous variants of hardware and software devices, as well as different versions of operating systems and firmware, testing all possible combinations of software and hardware may not be possible. Defining a reasonable subset that can be easily tested requires information to be gathered from end-users to understand which software versions and devices are being used by them. This information, then, needs to be analyzed to determine the most popular combinations.

Once it is known which operating systems and devices need to be tested, those combinations can be focused more on while less common combinations should be treated with smaller sanity tests.

A Bundle of IoT Communications Protocols to Test

IoT devices use various communication protocols to interact with each other, and with controllers. Protocols such as Constrained Application Protocol (CoAP), Extensible Messaging and Presence Protocol (XMPP), and Message Queuing Telemetry Transport (MQTT) are common, each of them having their own sets of pros and cons. But the most popular among them is MQTT, which can perform well in low bandwidth and high latency situations.

An API, typically based on XML or JavaScript Object Notation (JSON), is, most of the time, layered on top of the transport protocol to make it easier for developers to interact with the device. APIs and protocols can be used by testers for automated testing, and the test design depends on the protocols and APIs which are being used. Software testing and issue tracking tools must have the ability to support these APIs and protocols to be effective.

Emerging IoT Threats and Attack Surfaces

In October 2016, Dyn’s servers were on the receiving end of a huge distributed denial-of-service (DDOS) attack which brought down giants such as Spotify, Twitter, and PayPal among many others. The attackers were able to infect thousands of vulnerable IoT devices with Malware, causing devices to generate traffic.

It is said that more than 70% of the IoT devices used are currently vulnerable to security issues. With such a major risk, finding security loopholes before the attackers do so is critical. Special attention needs to be given to the password policy of the devices and it should be made sure that minimum password requirements are built into the device, and that they are enforced. It is also recommended to require a password change when the device is first accessed. This should be considered when automated tests are being developed.

Is the Diversity of IoT Applications and Devices a Challenge

Diversity is good, but it brings its own set of challenges. Like for testers, diversity means that they’ll need strong test capabilities to ensure that the performance remains consistently high across all devices and exceeds user expectations.

Without a robust test strategy, it is hard to understand the architecture and ensure that the software and devices being tested are always configured with the correct version. If the system is dependent on third-party services, then a change in third-party service can fail tests. This will be detected very quickly by automated tests that run as part of a continuous testing pipeline.

Unavailability of a third-party service leaves you with the option of virtualizing the service and removing the dependency on the actual service. The virtualized service can be used to test even if the service is down. A customizable service virtualization tool allows the configuration of expected responses from the service so that the application’s response to various situations can be tested.

Increased Load and Fast-Moving Data Pose a Separate Challenge

Connected IoT devices are dependent on fast communication. As a consequence, network quality can have a significant impact on the performance of the device. Smart devices are known to experience problems with network infrastructure such as inconsistent or slow internet connections, unreliable network hardware, and overburdened WiFi channels. To ensure that IoT applications and devices respond correctly without data loss, they must be tested across these different conditions.

Creating a network with all of the different conditions isn’t practical, to say the least, a lot of teams make use of network virtualization to programmatically emulate various conditions and responses. During testing, memory, CPU, and other device metrics should be monitored constantly.

Let’s take an example of an IoT device that has a memory leak. The device will perform slowly and may give incorrect responses. Software on laptop or desktop computers is activated and tested by user typing, clicking, or tapping. But the passiveness of IoT devices calls for testers to understand what devices are being used and how they behave. This requires an adjusted look at software testing tools that specialize in performance monitoring and testing as well as at issue tracking tools that can successfully help pinpoint the performance-related issues.

IoT Complexity Opens Door for Opportunity

The complexities of IoT we’ve discussed in terms of performance testing and monitoring are one side of the coin, the other side is the endless and exciting business opportunities that the devices present.

Testers need to adapt to new techniques and platforms to ensure that they successfully address the challenges IoT testing poses to deliver the best experience to the end-user in the shape of high-quality applications and devices. Fortunately, improvements to software testing and issue tracking tools, and infrastructure will continue to make the process easier over time.

Join the discussion