In August 2018 when we presented our research on the extreme vulnerability of many emergency services vehicles due to their use of onboard cellular gateways, we hoped to get the attention of people who could help change things. After all, when you tell the world you’ve been able to easily track police cruisers, in real-time, all over the country for the better part of two years, you expect people to pay attention. What we don’t know is if the right people are paying attention. In police and fire vehicles, ambulances, traffic signals, oil, gas, and water pipelines, and other applications, unsecured cellular gateways are exposing sensitive details to anyone who cares to see. From GPS coordinates to RADIUS secret keys, access to this information is only a default login/password away; some even display this information on an unauthenticated login screen. Just sitting there. For anyone to see.
Needless to say, something needs to change before people who don’t have good intentions decide to leverage this data for nefarious purposes. What we aim for now is accountability. Not any sort of vindictive accountability, but the kind of accountability we demand from our elected officials in a democratic society. Accountability to ensure that the men and women in emergency services whom we trust to enforce our laws, ensure public safety, and come to the rescue when things go wrong, are not at risk of falling victim to unsecured technology.
As researchers, our wild ride through the world of cellular gateways began in October of 2016. Somewhere along the way we discovered one of the gateways we were tracking was inside a police cruiser. Large numbers of others turned out to be the same, as well as in fire trucks and ambulances, and connected to traffic lights and security cameras. We never knew we would care so much about how the police cruiser we saw in traffic gained access to the Internet; or that we could pull up its GPS coordinates and follow it in real time.
We discovered we could not only track these vehicles in real time, but in precincts where officers took their vehicles home after their shift was over, we knew where they lived. We knew their routes to and from work, could watch as they responded to dispatch calls, and could learn their patrol patterns. We could use sensitive information in the device configuration to infiltrate the networks these devices connected to, and possibly manipulate data. In the wrong hands, the information could be deadly.
During the 22 months of research leading up to our presentation of this data at Black Hat 2018, over 100,000 devices were discovered globally, more than 13,500 disclosures were sent, and dozens of emergency services departments were found. The talk got the attention of industry professionals, but not the attention of those who managed these cellular gateways in use in their local police department’s cruisers. Chances are the people in charge of the fleets didn’t know their devices were vulnerable, or what risks they presented.
What’s wonderful about living in an open, democratic society is the availability of public records. As citizens, we want and deserve to know what our tax dollars are being spent on, and public documents provide a means to hold officials accountable. City councils and other municipal authorities often publish their records online, meaning it’s not hard to find who is using these devices and for what purpose. Disbursement approval records, city council meeting minutes, requests for bids, and the bids themselves have led us to dozens of specific municipalities using devices that may be vulnerable.
Some basic web scraping left us with tens of thousands of pages to review. The documents detail different municipalities’ moves to cellular gateways in their fleets, their traffic signals, water infrastructure, and environmental monitoring projects. We can piece together which devices they decided to use, for what purpose, and even which company won the contract.
For example, we have this receipt from a New Jersey company for a number of items, including 12 Sierra Wireless Airlink GX440 Wireless Routers and labor for the installation.1
(Note: F5 Labs has obfuscated all names in the following documents.)
Another interesting example is the City of Naples, Florida. Here, we came across a request for bids regarding the installation of 140 AVL (Automatic Vehicle Locator) units, including a Sierra Wireless AirLink GX400.1
If we dig a little deeper, we easily find that Naples City Council approved the purchase of 170 gateways in September of 2013.1
Conveniently linked from this document is the quote corresponding to the approval.1
We can even go watch the September 18, 2013 Naples City Council meeting where the Technology Services Director provides his brief overview of the items being purchased.1 During the meeting, he discloses that they tested the systems on police and fire vehicles over the summer and feel comfortable deploying them on all 200 vehicles, including marine. They further disclose they only plan to keep GPS records for their fleet for a max of one week, pointing to the challenges of storing large amounts of data since they collect GPS data every 5 seconds. Through these documents, we also learn which software they will be using to log their GPS data: ESRI ArcGIS GeoEvent Processor.2
Knowing they did a summer test run, we can go back and find the April 13, 2013 meeting minutes where the Technology Services Director stressed “that the purchase is critical as current equipment is simply not operating correctly.”1
That document links us to the memorandum showing those 17 Sierra Wireless AirLink GX440 devices are intended “for Naples Police Department’s Ford Interceptor Sedan Vehicles.”1
And, once again, we have a quote:1
It’s probably safe to assume these were part of the 30 vehicles referenced for the summer test.
Understandably, these devices purchased and installed in 2013/2014 are out of warranty and need to be replaced. But you don’t have to take our word for it, take the Naples Police Department’s word. In their FY 2018-19 CIP City Council Workshop presentation, they explain the need to purchase new gateways.1
This recommendation is backed up by a capital request contained in the May 14, 2018 City Council Workshop minutes for $31,000 to replace 37 gateways.1
The upgrade of these gateways presents yet another opportunity for these devices to not be properly secured upon deployment. Conspicuously missing from all of these meeting minutes and expenditure approvals is any mention of a security service for these wireless gateways. But, we don’t have to just hope they have taken the time to change the default user password and/or limited access to the management interface. We can begin reaching out to these people to ensure they have, or will.
The point of this is not to paint transparency in a negative light, but to show the ease with which a nefarious actor could use the tenets of our open, democratic society against us, reconnoitering vulnerable fleets to target.
As the months ticked by and the dots added up on our map, we did not disclose this information publicly, hoping to reach the right people through our disclosures. When all of them—over 13,500—went unanswered, we decided we had to speak up to get the attention of those who could help. Now it’s time to reach the people who can secure these devices upon which first responders rely. Like the City of Naples example above, many more cities and towns have purchased and installed these devices in their fleets. If not properly configured as we have previously shown, they could be vulnerable to network infiltration, data exfiltration, system manipulation, or violent attack.
As we outlined in our previous article, these “vulnerabilities” are not hardware or software faults. As long as firmware has been upgraded, administrators just need to change the default password, or create a new user and delete the default. Still, it’s a good idea to set up an ACL to limit access to only those who require access to the devices. This is not an insurmountable problem, it’s just a long-ignored one.
In the near future we will be releasing an article detailing all the municipalities for which we have found evidence that emergency and other workers may be using vulnerable equipment. The evidence for the report comes from public domain documentation produced by the officials for the location, and it is easily available to anyone who would choose to find it. The purpose of the report will be to enable concerned citizens to hold their officials accountable and ensure they secure the devices first responders rely upon to keep us all safe.
In the meantime, if your company is using these or any type of cellular or wireless IoT devices—here are some simple steps you can take immediately to protect yourselves from possible exploitation:
- Change the vendor’s default password, replacing it with one that is at least 16 characters long, if possible.
- Keep the devices updated with the manufacturer’s latest firmware.
- Restrict remote access to a specified management network by using IP tables or ACLs, or use a VPN tunnel if the product supports it.
- Avoid using default ports like telnet for remote administration; use SSH instead with keys, not passwords.
- Customize your devices configuration settings, paying special attention to what information is publicly available (don’t share GPS coordinates publicly to the entire internet), and log all events.
