#CrowdStrike Cause a Global Tech Outage - what happened, why, and (how) can it be prevented?

While the memes are amazingly good, and there's a lot of jest being spewed across the interwebs, this is a serious event with massive implications. So, in all seriousness, let's review the facts of the #CrowdStrike situation from 19-Jul-2024: 

As reported across global news outlets and the internets, a security company called CrowdStrike caused some chaos. There are cascading impacts across many industries. 

We are already seeing impacts: 

://courier service delays (UPS, FedEx, DHL, etc.) 

://flight delays/cancellations at the airport 

://small business closing for the day 

://websites being inaccessible 

://hospitals cancelling surgeries/treatments 

://municipalities being closed 

://government services being delayed 

among many other cascading effects that could last days, or weeks. 

While a major inconvenience, the bug was quickly resolved within CrowdStrike's system, so (as of publish date) the latest binaries are stable. Recovery will be slow and tedious, especially for larger networks, but the world will recover from this. 

What happened? As is being reported, a bug introduced during a routine update of their Falcon EDR software (anti-virus software run by millions and millions of customers) caused what is known as a kernel panic within the Windows operating system - we are seeing this manifest as a "bugcheck error" (aka - the Blue Screen Of Death , or #BSOD) on Windows machines. It does not affect #Apple or #Linux devices. Note: It is NOT a #Microsoft problem. 

How can we prevent this? Short answer, WE as users can't. However, this isn't the first time a large global tech vendor has caused major outages across the globe, and it won't be the last. 

How can CrowdStrike, or any another company, prevent this? Simply, adhering to the SDLC methodologies, adequate QA testing, and never do a full production roll out without fully testing in the field. A common practice is to deploy to 10% of the network and see how systems and users respond (yes sysadmins, you can do targeted deployments even if you don't have network segmentation in place). If all goes well, push to 25% and test again, then 50% and test again, then the full push. That way when a problem does occur, it doesn't take out everything and can be quickly fixed before a full production push. It's really IT Ops 101 - not that difficult. This is a good example of why you should backup your critical data frequently: whether to an external device, or a cloud storage facility (Google Drive, Dropbox, OneDrive, etc.). You should do this personally as often as you feel is necessary. Most companies have policies governing backup types, schedules, and testing methodologies. 

For my enterprise admins reading this, I hope you have a solid (and tested) backup methodology in place. Yes, you should test-restore your backups at least once per year, if not more often. If you can't restore the data, then what is the point of backing it up? 

So now the big question is, how does this issue get fixed? Well, it's a hands-on-machine fix (which means long days/nights and weekends for IT staffers for a bit). Since the devices are unable to boot, there's no back-of-house configuration that we admins can set to fix this. We literally have to put our hands on the device. The methodology is simple, and only takes about 5 minutes to do - but multiply that over hundreds, thousands, or even hundreds-of-thousands of devices and you can quickly see this is not a quick fix at scale. It is an even bigger nightmare for remote workers, who would need to be walked through the fix via telephone, making it a 30min fix (at best). In those cases, from my perspective, it makes more sense to send them a replacement machine that is not bricked, then reset the trouble device once back in hand. Hopefully you have the inventory ready and waiting, otherwise you need to grab a company credit card and hit up every electronics store in your city. What a fucking PITA. 

CrowdStrike's official guidance can be found on their webpage here: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ (external link). 

While all of this is happening, myself and most of my peers agree that CrowdStrike is still a quality vendor offering quality security products and services. This was just a BIG fuckup from whoever pushes out their updates. Clearly, someone did not follow protocol. 

As of this writing, CrowdStrike is the second largest security vendor in the world, which is why the impact of this was as massive as it was...and the cascade effect isn't done yet. There will be more fall out from this, not to mention the legal cases that could be brought against them in the aftermath due to the downtime. 

One of the biggest fallouts of this mess is phishing attacks - threat actors spinning up malicious domains claiming to fix the issue (they won't, they just want your money); emails being sent claiming to be able to fix the issue with "a click" (using a piggy-back technique to install a payload on your machine to do god knows what; oh and steal your money too). Please do not fall for the phish. It's won't end well for you, or your employer. 

There is no "easy button" here peeps. Just a massive Pain In The Ass. 

#StayCyberSecure 

#BeCyberAware

"Hackers are good. Infosec is evil."

I saw this comment while scrolling the interwebs and it struck a cord within me, being both a hacker and a professional in the infosec community. This comment is misleading and too absolute, I believe. 

Hackers are on both sides...good (white hat) and evil (black hat). Yes there are gray hats too, we'll get to that in a minute. 

Infosec is a discipline of hacking, relating specifically to security of data and systems. I cannot appreciate that it is inherently evil. What I know is that it's a commercialized discipline legitimizing hackers in society. They even offer college courses on it now, something I didn't have as an option! Infosec wouldn't exist if not for hackers. We wouldn't have firewalls, anti-virus software, encryption, or VPNs (among many, many other things), which are all designed to protect users and data from the bad guys AND users themselves. Yes, we users are our own worst enemy, but that's a story for another time. So tell me again infosec is evil, when its sole purpose is to, generally, do good by all netizens! 

People today are flocking to infosec jobs by the tens of thousands, which is great, cause we need them. Infosec brought hackers out of the shadows and into the light as white knights "saving the day", as it were. At the end of the day, which color hat you choose to wear is based on a very personal choice on morality and civility IMHO. Do you want to protect? Or attack? Do you want to help? Or cause chaos without remorse? It's a fine line, that's for sure, yet still a choice. 

Fundamentally, hacking is a positive thing! We look to advance technology and create digital systems in creative and imaginative ways. A core motivating value of our craft is: all information/data should be freely available to anyone who wants it, anywhere, at any time. Hard stop. Another core motivation is protecting the integrity of our digital history and not allowing any person or entity to censor information dissemination. Hard stop. Most importantly, protect humanz and human rights above all else. Hard stop. 

Yes, some individuals trend toward criminal thoughts and actions when processing these ideals, but they were already criminals with malicious intent who happen to use a computer, rather than a pistol. 

Most of us aren't criminals. 

Most of us are just kids who love electronics and technology so we learn everything we can about them. We physically take it apart, study every facet, and put it back together - sometimes even better than it was. We learn how to manipulate systems to our will. How to protect them. How to help with and foster innovation that advances and protects society. What breaks it and causes it to fail. How to "rejigger" it so, maybe, it doesn't fail. How to make a better version of what it was, or take the parts and pieces of the old to make something completely new. Perhaps our biggest responsibility is to mentor the next generation to not only appreciate where we've come from (our history), but especially what our fears are in the future. This isn't to scare them (though fear is a great motivator), it's to prepare them so they can become the hackers of the next generation - whatever that may look like. 

Society made some of the things we do illegal, IMHO out of fear. It doesn't stop us from fulfilling our core ideals. It's the interpretation of these ideals that make us inherently good or evil, at least in the eyes of society and to ourselves. 

Personally, I didn't realize I was a hacker, until I did lol. I started this game in the early 1980's as a literal child just trying to practice math and vocabulary words in a more fun way. My dad showed me how to find and edit source code of programs on our Tandy 1000. I added my school vocabulary words to a hangman game. I added my math homework to a some math program. I learned through computer programs I manipulated on a plastic box by pressing these small plastic squares. I was fascinated and excited. I learned better this way. The world seemed different now, but I didn't yet understand why. That came in time. 

I didn't know that was hacking. I don't even know if "hacking" had a real meaning back then (I was 5 lol). But here I am. 

I am confident that every digital advance we've seen in my lifetime can be accredited to hackers, which includes the totality of the Internet and space exploration (both inner and outer). The world would not be where it is today without hackers, good and bad. Infosec stemmed out of a societal need for protection of data and digital systems for humanz. Not only because of what the bad guys were actually doing, but also what the good guys theorized could happen. We hackers and crackers have, generally, the same level of expertise, just different motivations. 

Hacking shouldn't be a dirty word, but for a long time it was, and in some ways it still is. People and mass media commonly confuse a hacker with a cracker, which are not inclusive. I believe this is mostly mass-media's fault because they just don't understand. What's the difference? One is a criminal (cracker - short for "criminal hacker"), one is not. What makes the actions of a hacker criminal? Simply, when a law is broken. Hence the designations of white, gray, and black hats. A nod to the cowboy days of white and black hats: white is for the good guys, black for the bad guys - that made it easier for everyone to understand who was on which side in a fire fight. 

Gray is where most hackers and thereby infosec peeps live - we only have good intentions though sometimes we need to, technically, bend a law, or even break it, to accomplish our goal for the greater good. Again, our intentions are pure, but laws exist that make certain specific actions technically illegal. Hence why it's a "gray" area. Black hats are hardcore criminals whose only mission is to disrupt and/or steal, for financial gain, with complete disregard of any fall out - even if that results in the loss of life. 

White hats have a moral compass and good ethical beliefs, as do most gray hats. 

Black hats do not. 

The original definition of "hacker" I learned as a child, and still hold close to my heart today, went something like this: "an individual with advanced knowledge of computers and/or digital systems, who is capable of taking that system beyond it's pre-defined programmatic limits." So, basically, if someone makes any change on a system that goes bound the original programmed intent, that makes them a hacker too! For example, did you change the color theme and desktop background on your computer to a custom concept (not one of the canned choices)? You technically hacked the system. See, it's not all about writing malware, or attacking companies, or breaking into the government, or bringing down someone's website. It's about system manipulations in its purest, simplest form. 

So the next time someone semarily says that hackers or infosec are inherently good or evil, discuss their context. Approach it as a way to mentor or guide someone to a better appreciation of the craft, that is clearly not as black and white as anyone would have you believe. Help them understand that we just see the world differently than most. The euphoric streams of 1's and 0's, speeding alongside electrons, as they bounce everywhere and nowhere simultaneously, connecting humanz like nothing before, to everything. I think it's beautiful, in all of its glory - the good and the bad. It's more vast than our physical universe, but the size of a spec of space dust. 

I think one of the coolest things I realized in all my years is that at their true core digital systems and the internet are just electrons moving around and settling in different states in different physical locations. It's real, but not tangible. It's we hackers that have figured out how to manipulate those electrons into the world we live in today. The world most depend on to survive. Infosec is focused specifically on making the manipulations as safe as possible, for everyone. 

It is simultaneously good and evil. Both the greatest genius and greatest disappointment humanz have to offer at this moment in time. Respect it, don't fear it. Appreciate it, don't take it for granted. Be aware. Stay safe.

That's my perspective. This is my genius. 

I, am a hacker.

I know enough to make me dangerous. I know better than to be dangerous. I chose to protect, rather than to attack. 

How do you see things? What is your choice?