Looking back at 2020, it was obvious even at the time that everything had changed forever. The COVID-19 pandemic left nothing as it was. It brought disruption and loss to everyone. For security and IT staff, it also ushered in the Great Remote Access Experiment. Our work was suddenly thrust into the limelight, but without a shred of patience for the balance we’d carefully crafted. So, like everyone else, we improvised as best we could.
It's not that the virus and its effects are gone today; it’s just not a burning crisis anymore. We’ve learned how to manage things so we can live our lives productively. I suppose a bright side (or maybe a dark side) was that it helped us realize what a key role the Internet had in all our lives. Many learned how to do things online for the first time. Everyone—like, really, everyone—became familiar with video conferencing, telemedicine, and remote education.
With everyone under lockdown, the need for remote access grew at a staggering rate. To call it an explosion would hardly capture it. It wasn’t just the breadth of the need that shocked us, it was also the depth. In some cases, we had to put systems online that we never thought to make remotely available. There were some who didn’t need connectivity for work but instead needed it in lieu of work—to seek financial and psychological support, to try to stay connected with those they couldn’t see, or simply to forget for a little while.
We gave management the choice between doing it right or doing it quick. They chose quick. There was no time to talk about risk. We simply had to make it work, and we did. We recognized that we would not be able to reduce risk to a level anywhere near comfortable. Everyone just had to make some big guesses and work with what we had. In the midst of it, we knew there would be consequences.
So, we slammed remote access systems into production with none of the planning they need. We redesigned networks, repurposed hardware, and relaxed our rules. After nearly 20 years of debate about cloud computing, we finally had to jump into it like a soldier into a foxhole. We leaned on every remote access tool we had, whether it was properly encrypted or not. The richer IT shops had the resources to cope, but smaller organizations had to make do. But big or small, we were all in incident response mode, telling ourselves we’d clean it up later.
As the systems came online in their new forms, the magnitude of the changes became clear. Our data centers, then our workforce, quickly dispersed into the clouds and giant platforms that we had hesitated using for so long. The help desk completed its transition to metaphor after the desk was removed for decontamination. It also meant that more helpdesk services went to the lowest bidder, mostly offshore. Along with all of the new remote access capability went an enormous amount of monitoring and logging, even though the idea of spending time rummaging around in the logs seemed laughable.
Looking back, it’s clear we bought stuff we probably didn’t need. But we also bought stuff that saved our skin. We ended up using things in ways they were never meant to be used. Along the way, we learned a lot of new tricks. Such is the nature of emergencies and quantum leaps in innovation. DevOps, for instance, emerged in its final form as the best way for many of us to get things done in time.
Whenever we could, we tried to be careful about what we chose to deploy, made our best efforts to build for sustainability, and hoped we wouldn’t have to immediately tear it down. Once we hit on a solution that seemed like it would work, we deployed it fast and ruthlessly. If it failed, we rebuilt it until it held.
It was a new level of challenge. We learned to react quickly and improvise to fast-breaking events while at the same time considering the long-term consequences. The main lessons we learned from dealing with the complexity, uncertainty, and nonlinearity of the pandemic were:
- Isolate our most critical systems from anything we couldn’t absolutely verify each and every time they connected. They used to call it “zero trust” but now that everything is remote, it’s just how everyone does things now.
- Control internal interconnectivity by locking down, monitoring, and restricting connections to critical systems. That old fallen monarch, the firewall, became emperor again, but on the inside; not just facing the Internet.
- Simplify everything and shrink systems down to something we could manage from home with an overworked, highly distracted skeleton crew.
Throughout all of this we also had to communicate what was changing, but without adding to the anxiety. We warned executives, customers, patients, and partners about the ad-hoc nature of, well, everything. It wouldn’t be right to say that some of the changes we made were risky. All of the changes we made were risky. Once we got it all working, we were hesitant to adjust anything for fear it would fail. We made tradeoffs every hour and stacked compromise upon compromise. Some of the scariest risks we took involved things like:
- Scaling back strong authentication. The lockout rate for some users was too high.
- Slowing down patching even more. Patches broke things and we were already babysitting a house of cards.
- Keeping end-of-life equipment online. Conversely, we also paused upgrade and improvement projects, some indefinitely.
- Turning a blind eye to poor user security behavior. With all the new services and flood of new users, we knew some were writing passwords down or reusing them. Folks always do. When they did it in the office, we could remind them. At home, we just had to let it go.
- Allowing users’ personal systems and services to touch systems and data. There were more than we could even count but the users needed them. BYOD became SOP.
- Losing visibility. Even as we logged what we could, there was so much we couldn’t see. Between the BYOD, the home networks, and allowing split-tunnel VPNs, we simply couldn’t find or stop some threats until it was too late.
Indeed, we exposed a lot of things and inevitably, data were exposed and systems were disrupted by attackers. A significant (and surprising) sector of the cybercrime community resisted easy targets like unprotected hospitals, but not all of them did. Some cybercriminals even adapted to the new normal and produced a few new techniques, but most just repackaged their old hacks in the latest, COVID-branded dress.
Phishing campaigns switched lures to COVID-related flavors of clickbait, which were hard to spot and block as they flooded into our users’ inboxes. The explosion in ecommerce meant that existing vulnerabilities in web payment card forms were ruthlessly exploited. Remote access and teleconferencing systems were hit with every known password attack.
The various state-sponsored cyber threats that make up the APT pool experienced a brief dip in “service” while they got used to working remote, but quickly rallied with big disinformation campaigns to help mess up, among other things, the already messed-up elections, global trade in medical supplies, and to some extent the geopolitical order. They also invested considerable online effort pushing the blame around for the pandemic.
Some things in tech are unrecognizable from the pre-pandemic times, or as we say now, “the BC era.” The old business models became untenable, and entire industries disappeared or become unrecognizable overnight. Those industries that could subsist solely on online sales survived, even thrived in some cases, which contributed to the explosion in demand and strain on IT services. Some organizations, not just starving startups but also giants who had shaped the tech industry over the course of 30 years, were unable to adapt and fell along the way.
There were some victories. We proved that telemedicine can work, in the right context, and we found ways to share data about public health concerns while keeping some semblance of privacy across diverse systems, legal frameworks and cultures. The Great Remote Access Experiment represented a trial by fire for eLearning concepts and platforms that had been considered fringe before.
There were also signs of rebirth. The influx of tech experience and talent that flooded the job market along with all the other laid-off workers created a huge labor pool. This provided talent for startups, which are proliferating even faster now than they did in the 2010s. Some reputations were made in our industry, and some were destroyed. There was, however, a leveling effect in terms of demographics and geography. The permanent expansion in telework capability (as well as its normalization) meant that talent was less obligated to relocate to big tech cities. This, in turn, reduced salary variance, as it became less reasonable to demand Silicon Valley-caliber salaries when there were so many people everywhere able to do the work.
The fact that the Internet stayed as normal as it was under this extra technical and social burden was a testament to its design. We did what we could to give our customers the best and most secure technology support possible. But it was far from perfect, much less compliant. When they saw the mess, the auditors had kittens. In the end, though, they knew we had had no choice. As the lockdown eased and the new normal settled in, we patched and closed and fixed what we could, but huge structural changes like that can’t be easily reversed. It wasn’t as secure as we wanted but we hope it was secure enough. Maybe someday we’ll go back and reengineer things correctly. Who am I kidding? We’re going to live with it like we always do. So, we limp along forward into the future.
There are no blank slates, no fresh starts. The lessons we learned were paid for with economic, psychological, and technical debt that will take a generation to process. But next time we are not going to be so surprised. We owe it to ourselves and one another.
