Breaking Down the Door to Emergency Services through Cellular IoT Gateways

Note: This research was presented at Black Hat 2018.

Introduction

The Internet continues to evolve and expand at an astonishing rate. Our constant desire to remain connected or feel some sentiment of connection seems to be a high priority for individuals. But, cellular connectivity does not stop at consumers on a personal cell phone. The demand for constant connection extends into servicing nearly every industry that needs long-range, constant connectivity. This requirement is especially true of organizations that operate fleet vehicles, as innocuous as a delivery driver, or as critical as a police car.

Hollywood has provided a spectacular number of films depicting hackers involved in crime rings such as Lyle, the character portrayed by Seth Green in the Italian Job. At the end of the film, Lyle leverages his skills and talents to look after the health and welfare of his associates by manipulating traffic signals to control the flow of traffic, which subsequently assists in their successful heist.

This scene is no longer fantasy. For instance, the traffic lights that are referenced do exist. They are often connected back to a smart city’s infrastructure through the use of VPN tunnels and other private means of communication over devices like cellular gateways. These gateways are similar to the modems and routers used by consumers at home but with an additional feature, cellular connectivity, often in the form of 4G/LTE, if available. Additionally, these devices are capable of providing a variety of connection options, including wireless connectivity over 802.11x, Ethernet, USB, serial; analog and digital I/O; and cellular bands ranging from 2G through 4G LTE. If said devices are not configured properly, an attacker may be able to access them and do just as Lyle did in the Italian Job.

It feels like a time warp, but as with all cyber threats, they do not appear instantly. They evolve slowly in the background over long periods of time until the problem seems to reach a critical mass. Many threats grow over the course of months and years before anyone notices—if they notice at all. It is highly likely that if a threat actor has laid eyes on a target and decided to begin attacking, discovering the attack (let alone thwarting it) could be a real headache, given the lack of visibility into the what, who, where, and when in logging activity.

Critical emergency services such as police, fire, and medical manage their fleets with vulnerable cellular IoT devices. “Vulnerable” doesn’t have to mean a vulnerability within the hardware or software, although we suspect that is the case in some makes and models, we just haven’t tested every model or manufacturer uncovered. In this instance, we are talking about devices susceptible to remote attacks because of their weak access control and use of default credentials. Once accessed, an attacker can use the device to launch attacks, as we have seen with thingbots like Mirai and Reaper, or they can use that access for nefarious purposes to spy, redirect commands in the case of a fleet taking orders from a remote command, or shut the system off, effectively disabling operations. Because the critical emergency services we depend upon are using these systems, this is an urgent human-interest matter as we have moved beyond the unpleasant life impact of stolen data, and into attacks that can literally cost people their lives.

Nearly two years have passed since we first started observing cellular gateways distributing packets across the internet. Today, we are only scratching the surface of what will inevitably turn into years of future research and discoveries before the world has tackled the problem of IoT devices being deployed without security considerations. For now, this article includes the following, and will be followed up with future research and discoveries.

• The existence of cellular IoT devices that are not properly configured is allowing attackers to easily leverage remote administration for nefarious purposes.
  • The improperly configured devices we discovered and tested had either default administration credentials (such as admin:12345), or they required no authentication at all.
• The absence of logging capabilities on these devices ensures that nefarious activities cannot be tracked.
• Because most of the use cases for cellular IoT are for moving fleets, devices that need tracking, or remote critical infrastructure, virtually all of them have GPS coordinates. Excessive information disclosure, such as providing GPS coordinates publicly without requiring authentication (as some devices we discovered do) is giving attackers the ability to track fleet vehicles without ever breaking the law with unauthorized access. Yes, police cars can be tracked without breaking the law.
• There is no bias on which industries or cellular device manufacturer will fall victim to threats emerging from cellular devices. Virtually every industry that requires some form of long-range, constant connectivity is impacted (and likely, most manufacturers) as development standards apply unilaterally.
• As of July 28, 2018, we have identified more than 100,000 devices that are impacted online. 86% of the devices identified exist within the United States.
• Attackers have been exploiting many of these systems since August 2016, if not earlier.
• We have a defined list of impacted Sierra Wireless makes and models, however, we believe the problem to be widespread across all manufacturers of cellular IoT devices.

Discovery

Discovering anything new is often an accident; we stumble or trip and fall into a finding that wasn’t the initial intent, as was the case for this research. This research unintentionally began while looking into Bashlite.¹ The Bashlite research just so happened to coincide with the Mirai attack on DynDNS. While looking into the Bashlite incident, we stumbled upon a device we originally thought was a DVR infected by Bashlite. Looking further, we identified it as a digital signage device responsible for displaying flight arrival and departure times at a major airport in Europe. The system was actively compromised, which we disclosed to the impacted party. That party shipped the infected drives to us, which ultimately served as ground zero for this body of research. During the Bashlite incident, we were wondering how big the problem might be, so we began scanning. First manually on October 22, 2016, and then using a more automated approach starting on October 24, 2016. Part of that scanning effort included the airport network impacted by the Bashlite incident as they gave us authorization to do so. Those scans revealed cellular gateways manufactured by Sierra Wireless, including one within the airport network that was publicly accessible.

Initially we used Shodan to get an estimate of how many Sierra Wireless gateways exist on the Internet, and it gave us a preview of what we anticipated: a lengthy list of hostnames and IP addresses to research further. Sierra itself describes how vast and deep its reach is within the IOT and industrial system space—the industrial controls that are often threat-modeled to not include connectivity to the Internet. They are often highly disparate networks of devices that do not “speak IP” like packet switching networks do.

What did we find?

In October of 2016 we identified 49,962 hosts that appeared to be vulnerable to weak authentication in use by Sierra devices. The vast majority of these devices were located within the United States (84%), and therefore the US became our initial focus. The map in Figure 1 shows the GPS coordinates of the public-facing static IP addresses of devices discovered in the US in October 2016.

We continued scanning from October 24, 2016 through July 30, 2018. Each scan became more precise as we tuned our scans connecting the dots between the affected models and software versions through SSL certificates, hashes of image assets in use on Sierra’s configuration GUI, and copyright references in their footer.

In September 2017, we discovered 58,670 hosts. During this time we were also attempting to tune our scans through fingerprinting devices. This effort proved difficult because the devices would often disappear.

In July 2018, we discovered 105,400 hosts, as shown in Figure 2. This figure also compares Shodan results (the search engine we initially started with to discover devices), with scan data from our partner Loryka, because they were able to find a significantly larger number of devices than Shodan. Despite the larger numbers, the spread of devices across networks remained steady with approximately 70% of devices living on Verizon’s network.

Figure 3 below shows the breakdown of Sierra Wireless models and their prevalence.

From October 2016 to July 2018, we identified nearly 50,000 additional hosts globally. This increase is due not just to the nature of growth in this emerging industry but also to the methods we used to improve identifying devices.

The map in Figure 5 shows the GPS coordinates of impacted devices in July 2018. Eighty-four percent of the devices exist in the US, however, it’s apparent that cellular devices are starting to be used all over the world.

Disclosure Attempt

In October 2016, 417 disclosures were sent, but zero replies were received. Initially, we held off public disclosures, but due to the potential impact to human life, and because the problem has been largely ignored by the impacted entities (those that received disclosure notifications), we finally decided to talk about the research publicly in an effort to get the problem resolved. Once our talk about this research was announced by Black Hat, we began to see results from our disclosures, although still very few. By July 2018, over 13,500 disclosures had been sent, but we still only had two replies. One of those replies was from Sierra Wireless, which began a conversation that would help us refine our scans and present a more accurate picture of the problem at hand.

Research

Following our scans, we began performing research on manufacturers, the configuration GUIs, identifying fleet vehicles, mucking with GPS, toying with radios, discovering other manufacturers, reviewing configuration templates, and logging configurations.

We discovered a Sierra Wireless bulletin (as shown in Figure 6) warning of Mirai infections.²

The Sierra notice also included a list of impacted makes and models.³

In addition, on October 12, 2016, the US CERT issued an alert (ICS-ALERT-16-286-01)⁴ for industrial control systems warning about Mirai infections on Sierra Wireless devices (see Figure 8).

Publicly Available Discoveries

Most vendors champion their customers on their websites through case studies, and Sierra Wireless is no different. Their website provided a wealth of customer stories and details about the types of applications many customers were using Sierra devices for:

• St. John Ambulance, Western Australia
• California Highway Patrol, California
• Ventura County Fire Department, California
• South Bay Regional Public Communications Authority (SBRPCA), California
• Los Angeles County Sheriff’s Department (LASD). California
• West Metro Fire Protection District, Colorado
• Westminster Police Department, Colorado
• Danish National Police, Denmark
• Acadian Ambulance Service, Louisiana & Texas
• East Baton Rouge Parish Emergency Medical Services (EMS), Louisiana
• Mississippi Highway Safety Patrol
• Gem Ambulance, New Jersey
• Charlotte-Mecklenburg Police Department, North Carolina
• City of Charlotte, North Carolina
• Brazos County Sheriff’s Office, Texas
• Harris County, Texas
• Dickinson Police Department (DPD), Texas
• Fairfax’s Urban Search and Rescue Team, Virginia
• South Wales Police, Wales
• City of Yakima, Washington
• Seattle Fire Department, Washington

We matched up the Sierra Wireless devices discovered in our scans with known customers from Sierra Wireless’s website and overlaid that onto the 300 most populous cities in the US.

Sierra Wireless wasn’t the only manufacturer we discovered in our research. Moxa, DIGI, and Cambium also came up, however, they didn’t represent a large portion of our scan results and we did not purchase any of their devices for testing in a lab. Therefore, they are not included in this research piece, however, we do plan to test them in future research projects.

Lab Setup

To explore all the potential presented by this “vulnerability,” we set up a lab with some of the most common devices we found with our scans. Our lab setup consisted of the GX440, ES300, and the GX450.

We configured the devices in a similar way to how we had seen them “in the wild” during our scans. The following screenshots are taken from Sierra Wireless devices we tested in our lab environment.

Exploiting the Vulnerability

As mentioned earlier, “exploiting” these devices is not done through a typical hardware or software vulnerability. There is no weakness in the software to exploit. There’s no hacking of the hardware. This is a weak admin user authentication exploit—the age-old “user didn’t change the default password” story.

Figure 11 is a screen shot of the Sierra Wireless ACEmanager admin interface. The admin username is “admin” and the password is “12345.”

We also noticed that several users deploying these devices were turning on options to show location data on the login page of the configuration software, including the latitude and longitude of the device.

A simple Google search of the latitude and longitude coordinates made it easier for us to see what kinds of applications we were finding.

Some device owners even chose to disclose more connection data, including their cellular signal information. Even if they chose not to disclose their latitude/longitude, knowing their LAC (Location Area Code) and the CELLID enables us to get an approximate location of the device. The LAC gives us the carrier of the cell tower they are connected to, and the CELLID defines what tower they are connected to.

Configuration GUI

The screenshots in Figures 14 – 18 show the configuration GUI from login, to status page, and then how incredibly easy it is to change the vendor device admin password.

This status page is presented once authenticated into the device.

The admin password can simply be changed in the admin tab.

The configuration is available to download as an XML template, which includes sensitive data in plain text such as the IPSec tunnel secret, RADIUS, TACACS, and WPA2, if selected.

We wrote a script that serves as a template crawler, essentially to enumerate through our lab devices, authenticate, and download the configuration template. We then parsed these configurations to identify strings of interest.

There is quite a lot of information to be read within the configuration. As a user of the device within our lab, that is not much of a concern. However, if a nefarious actor were to gain access and download configurations, it could be another storyline altogether.

Figure 19 shows the available logging options provided by Sierra. Most of the useful logs are turned off by default. This allows for attackers to compromise these systems without leaving a trace.

In our lab, we made a change to the device, and, as shown in Figure 20, there is no useful information within the log file to indicate we made a connection to the device and made any sort of change.

Pattern Discovered—They are Fleet Vehicles!

As our lab research continued, so did our scanning and research into the devices we had discovered. We noticed a pattern within our scans that indicated some of the devices (those with static IP addresses) were available to us, and then they would seemingly disappear. These devices would come online at specific times, would occasionally go offline throughout a several hour period, then go offline for 8-12 hours. What we began to realize is that these resembled workers’ hourly shifts.

We then returned to the publicly facing latitude and longitude data we had seen all throughout or research. We began to parse out the GPS data into Traccar, an open source fleet tracking software, and realized we were, in fact, tracking vehicles on the road. These were fleet vehicles moving around US cities.

So, we developed our own simple JavaScript application to visualize what we were collecting from publicly available information as shown in Figure 22 below.

Potential Attacks and Impact

We stumbled upon the issues with cellular IoT devices during our “Hunt for IoT” research of devices that were infected by Mirai. Attackers know how to exploit these systems and are actively monitoring them. Sierra Wireless, one of the largest manufacturers of cellular IoT devices, issued a public statement describing how to secure their devices from Mirai infections. US intelligence agencies have made public the fact that Russia attacks IoT devices to maintain long-term, persistent access for future offensive purposes. It is not a stretch of anyone’s imagination that Russia—or any other adversarial nation-state—could compromise these devices, as well, and use them to their own advantage. But this isn’t just a US issue. These devices are deployed in the same types of systems around the world and can absolutely be used in warfare, terrorist attacks, or by lone actors targeting specific people or businesses.

If a system requires long-range constant connectivity, it’s important. Often critically important. We found many systems used to support human life such as police cars, fire trucks and ambulance fleets that are impacted by this research. Outside of what we already know attackers are using these systems for (such as relaying attacks when the devices are infected with Mirai), they could be used by nefarious hackers, criminals, nation-states, or terrorist groups to:

• Maintain persistent access to spy on people, businesses and operations
• Maintain persistent access to surveil and monitor law enforcement to aid in evasion
• Maintain access to monitor law enforcement activity to aid in ongoing litigation
• Gain control of device commands and redirect, reroute, or delete.
• In warfare or terrorist activities:
  • Take offline, effectively disabling the system relying on the cellular modem for operations
  • Disrupt the physical activities of a system relying upon commands from the cellular modem.
  • Disrupt communications on a wide scale by attacking the entire fleet at once.

“1-800-CRIME-AID”

If we momentarily transport ourselves into the land of television and movies—the land where bad guys have extraordinary abilities to command traffic signals and hijack surveillance systems—we find the scenarios this type of vulnerability opens up.

Infrastructure needs connectivity. Wireless gateways are useful for this. Attached to traffic cameras and traffic lights, they provide municipal workers up-to-the-minute data on and more efficient management of their systems. At the same time, when improperly configured (as we have seen in large numbers), they present an opportunity for an adversarial actor to manipulate the information delivered via the gateway, or even the systems to which they provide access.

Combine that with real-time GPS tracking of a police department’s cruiser fleet and attackers have several of the tools necessary to facilitate a security-system-bypassing, camera-freezing, police-tracking, traffic light-changing heist à la The Italian Job.

Striking the Sentinels

The possibilities quickly took a dark turn. In late 2016 and early 2017, when our research was in its early stages, it was national news that 2016 was an especially fatal year for law enforcement officers. In fact, ambush-style assassinations on police officers were up over 150%. With this dominating much of the national discourse at that time, it wasn’t difficult to see the dangers posed by being able to track police officers in their vehicles, in real time. It is a horrifying thought, and we’ll leave it at that, but a real possibility.

Request Timed Out

Another horrifying yet real scenario involves medical help never making it to the scene, or at least not in any condition to provide aid. It is a tactic that began to emerge as terrorist attacks evolved over the past two decades. Emergency medical personnel en route to save lives become victims themselves, increasing the time before initial victims received help and reducing overall emergency response capacity. The same way police cruisers can be tracked, so can ambulances. The same way police cruisers can be tracked, so can ambulances.

Distributed Denial of Democracy

At this point, it should be crystal clear the dangerous doors this problem opens. And if we take a moment to revisit APTs, and if we’ve been paying attention over the last two years, we know there are concerns about election security and integrity. Disruption of a single emergency call is dangerous enough to merit attention. Surreptitious manipulation of traffic signals combined with disruption of police, fire, and medical systems to suppress voter turnout in key districts merits shouting from the rooftops. It is personal safety as well as a national security issue.

Vigilantes might want to take matters into their own hands like their efforts we saw with Mirai and BrickerBot, however they cannot do this without causing damage to systems that support human life. Vigilante efforts will ultimately hurt people, so they cannot participate in this cleanup. The best-case scenario right now (from a third-party remote access standpoint), is a white hat discovering a vulnerable system and not breaking the law by doing nothing, or breaking the law in an effort to help by changing the default admin password and upgrading the firmware.

This goes beyond DDoS and beyond the exfiltration of data (even though that is a major concern). The second- and third-order effects of these wireless gateways being compromised mean that not only can sensitive information on police, medical, and other networks be exfiltrated, but the systems that first responders rely on to provide life-saving services can be disrupted or otherwise abused.

We’re talking long-range, constant connectivity to critical infrastructure that is used to support human life or can directly impact lives if taken out or otherwise compromised. Police, fire, medical, utility and other municipal fleets impacted, traffic cameras and signal lights compromised.

Take the attack scenarios presented into consideration with the industries and services we know use these types of devices. The Black Hat presentation focused on police cars, ambulances and fire trucks as examples, but these systems are used across virtually every industry that has fleets that need to be managed, or critical systems that need monitoring and constant connectivity. Therefore, the potential impact is big. We have yet to find an industry that isn’t impacted. The following list of potentially impacted industries is what we gathered from simple Google searches:

• Service Providers that provide cellular connectivity to these devices. Verizon is the biggest, followed by AT&T, Telestra, Sprint PCS, Telefonica Spain, Orange Israel, Bell Mobility, Orange France, Telus Mobility, and Rogers Wireless.
• Federal and Local City Governments and Law Enforcement. Many cellular IoT manufacturers champion their customers and use cases on their websites. Sierra Wireless is just one example, which we pointed out earlier in this article.
• Financial. ATMs are a consistent use case on most cellular IoT manufactures websites.
• Retail. Point-of-sale (POS) systems and kiosks are consistently referenced as use cases on cellular IoT manufacturers’ websites.
• Mining, Fossil Fuels, Energy (Refueling Stations including and Hydrogen Refueling Stations), Maritime, Shipping, Transportation, Utility, Hospitality, Digital Signage, and Robotics Industries. All of these industries need long-range, constant connectivity and, in most cases, fleet management, to maintain operational efficiency. As such, all of these industries are often championed as use cases for cellular IoT on manufacturers’ websites.
• Construction. Large construction sites need Internet connectivity for communications before physical lines are laid.
• Broadcasting. Uninterrupted live coverage requires multiple cellular IoT devices, so the broadcasting industry is often championed as a use case on cellular IoT on manufacturers’ websites.

Remediation

Failure to harden systems that results in remote access compromises is a widespread problem across IoT devices. This report has focused on Sierra Wireless devices, simply because they make up 80% of the cellular IoT market share and therefore appeared most often in our scanning. But the lessons learned and remediation activities apply to all IoT devices whether cellular or wireless. Here are steps every user of these devices should take immediately:

1. Change the admin password. Because brute force attacks can still occur, make your admin password as long as possible, preferably 16 characters or more.
2. If possible, use ACLs to restrict remote access to a specified management network or leverage VPN tunneling, which is an option with many of these devices.
3. Stay up to date with the device’s latest firmware.
4. Don’t use default ports, especially telnet, for remote administration! Shift to SSH and use keys instead of passwords.
5. Update your device configuration settings. At a minimum:
   1. Log events! (Log retention capacity limits may apply.)
   2. Don’t share GPS coordinates or any identifiable information publicly if you can avoid it. In the case of cellular IoT, improper information disclosure could literally put lives in jeopardy.

For purposes of this report, Sierra Wireless has graciously offered to assist customers securing their devices. If you need assistance with any of the items above, reach out to security@sierrawireless.com.

If you are a developer of IoT in general, especially cellular IoT products that are implemented to support critical systems, build in a requirement to reset the administration password upon initial login. Configuration settings should also be set to not disclose confidential information by default. Any settings that makes the device inherently less secure should be configurable by the device owner rather than being set by default.

Conclusion

Researching cyber threats is a critical service to global citizens simply because the manufacturers making these products don’t consider potential future cyber threats into their models. Add the human element (device administration errors will happen) and the reality that nefarious threat actors will eventually find every insecure system on the Internet, it is only a matter of time before a catastrophic cyber attack occurs that leverages cellular gateways. The time to act is now, especially with simple remediation steps to solve already present problems. And with common-sense solutions built into products by vendors, we can help keep weak credentials from having catastrophic consequences.

Please share this story far and wide so action is taken globally, because over 13,500 disclosures have been sent, yet there still have only been two replies. If it weren’t for white hat researchers, we would be finding out about discoveries like this from news media after a terror attack, which is entirely too late. The right thing to do is avoid the risk by remediating now.