#CrowdStrike Cause a Global Tech Outage - what happened, why, and (how) can it be prevented?

While the memes are amazingly good, and there's a lot of jest being spewed across the interwebs, this is a serious event with massive implications. So, in all seriousness, let's review the facts of the #CrowdStrike situation from 19-Jul-2024: 

As reported across global news outlets and the internets, a security company called CrowdStrike caused some chaos. There are cascading impacts across many industries. 

We are already seeing impacts: 

://courier service delays (UPS, FedEx, DHL, etc.) 

://flight delays/cancellations at the airport 

://small business closing for the day 

://websites being inaccessible 

://hospitals cancelling surgeries/treatments 

://municipalities being closed 

://government services being delayed 

among many other cascading effects that could last days, or weeks. 

While a major inconvenience, the bug was quickly resolved within CrowdStrike's system, so (as of publish date) the latest binaries are stable. Recovery will be slow and tedious, especially for larger networks, but the world will recover from this. 

What happened? As is being reported, a bug introduced during a routine update of their Falcon EDR software (anti-virus software run by millions and millions of customers) caused what is known as a kernel panic within the Windows operating system - we are seeing this manifest as a "bugcheck error" (aka - the Blue Screen Of Death , or #BSOD) on Windows machines. It does not affect #Apple or #Linux devices. Note: It is NOT a #Microsoft problem. 

How can we prevent this? Short answer, WE as users can't. However, this isn't the first time a large global tech vendor has caused major outages across the globe, and it won't be the last. 

How can CrowdStrike, or any another company, prevent this? Simply, adhering to the SDLC methodologies, adequate QA testing, and never do a full production roll out without fully testing in the field. A common practice is to deploy to 10% of the network and see how systems and users respond (yes sysadmins, you can do targeted deployments even if you don't have network segmentation in place). If all goes well, push to 25% and test again, then 50% and test again, then the full push. That way when a problem does occur, it doesn't take out everything and can be quickly fixed before a full production push. It's really IT Ops 101 - not that difficult. This is a good example of why you should backup your critical data frequently: whether to an external device, or a cloud storage facility (Google Drive, Dropbox, OneDrive, etc.). You should do this personally as often as you feel is necessary. Most companies have policies governing backup types, schedules, and testing methodologies. 

For my enterprise admins reading this, I hope you have a solid (and tested) backup methodology in place. Yes, you should test-restore your backups at least once per year, if not more often. If you can't restore the data, then what is the point of backing it up? 

So now the big question is, how does this issue get fixed? Well, it's a hands-on-machine fix (which means long days/nights and weekends for IT staffers for a bit). Since the devices are unable to boot, there's no back-of-house configuration that we admins can set to fix this. We literally have to put our hands on the device. The methodology is simple, and only takes about 5 minutes to do - but multiply that over hundreds, thousands, or even hundreds-of-thousands of devices and you can quickly see this is not a quick fix at scale. It is an even bigger nightmare for remote workers, who would need to be walked through the fix via telephone, making it a 30min fix (at best). In those cases, from my perspective, it makes more sense to send them a replacement machine that is not bricked, then reset the trouble device once back in hand. Hopefully you have the inventory ready and waiting, otherwise you need to grab a company credit card and hit up every electronics store in your city. What a fucking PITA. 

CrowdStrike's official guidance can be found on their webpage here: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ (external link). 

While all of this is happening, myself and most of my peers agree that CrowdStrike is still a quality vendor offering quality security products and services. This was just a BIG fuckup from whoever pushes out their updates. Clearly, someone did not follow protocol. 

As of this writing, CrowdStrike is the second largest security vendor in the world, which is why the impact of this was as massive as it was...and the cascade effect isn't done yet. There will be more fall out from this, not to mention the legal cases that could be brought against them in the aftermath due to the downtime. 

One of the biggest fallouts of this mess is phishing attacks - threat actors spinning up malicious domains claiming to fix the issue (they won't, they just want your money); emails being sent claiming to be able to fix the issue with "a click" (using a piggy-back technique to install a payload on your machine to do god knows what; oh and steal your money too). Please do not fall for the phish. It's won't end well for you, or your employer. 

There is no "easy button" here peeps. Just a massive Pain In The Ass. 

#StayCyberSecure 

#BeCyberAware

Time to Air Gap

In a world of 24 hour activity literally being streamed in real time across the globe and beyond at a rate of trillions of bytes per second at the speed of light, the biggest question in privacy is how to achieve anonymity in a world where almost nothing is secret. There is still a way to go 99%. It's simpler than you think, but an effort nonetheless.

Some background. Everything we do in digital form is cataloged and stored in a vast array of databases and servers across an amazing amount of touch points, which is then synchronized across a thousand other servers for redundancy and caching, which is then backed up to dozens of other servers, with their own redundant backups. Anything you put online...any application you use...any "terms of service" you agree to...any text or media you post...remains online forever. With the right tools and search terms, anything can be searched for, or spyed on, or downloaded in an instant. It's been this way for decades, and will continue to be that way for centuries to come, especially with as connected as the planet is and as long as there is electricity.

Some discussion. Cyber attacks are a constant thing. Increasingly, we should take as a starting point that cybersecurity compromises are the third certainty in life. The cyber world is constantly at war with itself. Governments hacking governments. Corporations hacking corporations. Governments hacking corporations. Hackers hacking governments and corporations. Hackers hacking hackers. Governments and corporations hacking hackers. And then there's everyone else. Generally oblivious. Privacy is a luxury, which we give up willingly every single second of every day. The emergence of intelligent systems, artifical neural networks, and deep thinking algorithms only proliferate this further. They take, store, and learn from every bit of data we leave as breadcrumbs. Artificial intelligence is here, and it is learning. From us. And we're letting it. Give it enough processing power, and it becomes self aware. Quantum computers will make that very real, very soon.

Some perspective. Having lived through the evolution of modern computing, including the Internet, all of this is absolutely fucking amazing, and a geek's ultimate wet dream. A demonstration of true humanz genius, ingenuity, and progress (not as far as we should be, but progress nonetheless). Highly impressive in the vastness of its brilliance and simple complexity. I Iove using It, and learning about It, and protecting It. All of it, if I am completely honest, scares the living shit out of me. There is too much. It has become frightening. AI is now making decisions and inferences faster than humans, and has even been seen generating its own programming code. So, the concept of air gapping entered my mind as a way to keep safer than I already am. Most cannot see the signs, or do not want to admit they exist, however I am of the firm belief that World War III has been well underway, and we need to protect ourselves, especially our digital lives as I feel they are the most vulnerable to compromise. Stay with me, it's all relevant.

It has been discussed for decades that the next major global war would be fought half online, and half in the real world. The evidence is all there, and I do not believe it to be simple coincidence. Global newz outlets, small town newz papers, radio ztations, and zocial media have been propagating images of this war. Pick a topic...WMD's, genocide, terrorism, ransomware, deep web market hackz and seizures, arrests of crackers and phreakz, data breaches, RFID implants, cyber surveillance initiatives, counter cyber terrorism, weapons trafficking, the unavailability of bullets to the public, gun control politics, powerful botnetz, election hacks, political hackz, hardened/weaponized computer systems...I hope you get the point.

Back full circle. Traditionally, air gapping a system means it doesn't have any network interface cards, or external drives with which to access or extract the data contained within said system. You can not get close enough to implant a listening device that reads vibrations or thermal changes being given off by the system's internal hardware to convert that into bits representing the data being actively accessed (such as login credentials, encryption/decryption key exchanges, data manipulations, etc.). The only way to extract the data contained within is by sitting at the console and physically removing the locked and encrypted drives, if there is no SD port. Then, if you can pull the impossible off (which includes getting the data off campus), you would need supercomputer power to decrypt the contents of the drive, which would still take 1,000 years to break (if and unless you are lucky). There is still the idea that once you decrypt the data, it could transmit its location to its owner, meaning you too would need an air gapped system to exfilitrate the data. Then comes what you do with said data. Yet another  catch 22. The NSA, CIA, FBI, DEA, DHS, militaries, every government, and super corporations maintain their most secret data on air gapped systems. Physical access to these systems is extremely limited and highly controlled. It's considered the safest digital platform because the system isn't connected to anything but a power cable, and thus, in theory, cannot be hacked. A true digital safe, as it were. We know anything can be hacked, it just takes time. As a hacker, we count on human error and complacency, making even air gapping a 99% solution, and the best we've got. Now, take this concept and apply it to a human life. It's far simpler, and also 99%. The anomaly, is human nature.

Based on my research, here is what I have learned about how to go 99% off grid digitally. While I do not yet practice everything I note here, I am closer than even those who know me best are even aware.

1) Get off the internet, period. No social media, no surfing the clearnet, no online purchases, no clearnet email accounts. If it becomes absolutely necessary to access the Internet for a specific purpose, there are completely anonymous ways to do these tasks, on secure systems like Tails over TOR, for example, using cryptocurrency, and ghost mailboxes. Avoid Google at all costs. Use TOR browser, responsibly (www.torproject.org). But generally, just leave it all. Stop posting immediately, delete your accounts, and never go back.
2) Get rid of your smartphones, tablets, Windows and Apple computers, smart devices (TV's and refrigerators included), iRobots, etc. Need a cell phone? Buy a prepaid flip phone, and change it (and the number) every month (aka burners). Every phone can eventually be traced and tracked. Still need a computer? Learn Linux, how to secure it, and practice way smarter browsing habits (use TOR browser), if you browse at all. Keep in touch with world events, anonymously, and continuously hone your skills.
3) Always use cash or cryptocurrency, for everything. If you have to make an online purchase, use cryptocurrency, the deep web (local markets only, don't buy overseas, and be very careful), and have it shipped somewhere that is not your home, like a post office box, a business, or an associate's location, under a false name. By the way, there are ATM's now that you can convert cash to BTC, and visa versa. Look it up using Duckduckgo.com (a safe search engine).
4) Drive an older car that does not have a computer in it, or at least has all analog systems. Yes, cars are also being hacked, remotely. Keep it clean and running well though, you don't want to draw undue attention. Walk or take public transportation when you can, avoiding direct face contact with cameras. When you do go places, change your entry/exit routes regularly...avoid habitual patterns, unless necessary to remain hidden in plain site (like going to work, or getting groceries).
5) If you must, have an immaculate and purposeful digital/public footprint. Which means a clean record, and a "normal" looking life, so as to not draw undue attention. Keep it super minimal and protected, even fake some details if you wish, but it has to be believable. Your outward personality must seem conforming, friendly, and genuine. When people search for you online, they need to find only what you want them to find. Purposeful is the key word here. To keep your accounts secure, use a dice word list to generate passphrases with an entropy of 7 to 10 or more words (as the host allows), and rotate passwords on a schedule.
6) Second most important after getting offline, and the best to mention as the final advice, would be live simple and minimalistic. Only get what you literally need to live comfortably, and look "normal". The trick about hiding in plain site is being distant enough that people respect your privacy, but involved enough that they believe you to be a "normal, nice guy/gal". Avoid run-ins with the law and reporters. Do not have public arguments. Remain intelligent, articulate, empathetic, determined, and most of all inquisitive. Question anything, be aware of everything.

If you can literally get out of dodge and move to the mountains in the middle of nowhere, or something like that, the closer to 99% you get. If you are not online, there is nothing to take/attack. Here again, human nature is the anomaly.

You can be connected, yet a ghost. You can see the world, without a face. You can reach out, without being reachable. The less connected you can maintain, the better. I am committed. How far are you willing to go?

~Geek

This blog is only to express the opinions of the creator.  Inline tags above link to external sites to further your understanding of current methods and/or technologies in use, or to clarify meaning of certain technical terms.  Any copyrighted or trademarked terms or abbreviations are used for educational purposes and remain the sole property of their respective owners.

brought to you by http://geekofthehouse.blogspot.com

Competitive Forces that Shape IT Strategy in Business

Competitive Forces

The introduction of information technology (IT) systems has changed how companies conduct business, and also how they compete in their respective markets. There are a number of risks and advantages to implementing an IT system, which can be managed with the correct mix of technologies as an integrated platform. The purpose of this paper is to review the competitive forces that shape IT strategy in business.

IT Risk to Competitive Advantage

One of the primary risks to a company’s competitive advantages is systems availability. The computer has become a key tool in the art of conducting business, which means that they must be reliable and provide the resources necessary for a person to meet or exceed the expectation of their role. From an IT perspective, system failure is something that should be proactively monitored across the enterprise so that downtime is as minimal as possible in nearly all potential scenarios. The loss of revenues from being offline can be multiples higher with companies that provide 24/7 services to their clients, where revenues are calculated by the minute.

Another risk to competitive advantage is the disclosure of sensitive or proprietary data that is the source of the company’s advantage. A sales agencies value to a manufacturer, for example, derives from its industry contacts and distribution network. Therefore, their contact databases become their most valuable asset. A risk is espionage, an insider could provide these details to a competitor, or to a manufacturer looking to cause disruption in the market by selling online or via direct sales. Another risk in the disclosure of sensitive data that represents customer’s private information, including contact information and financial transaction data. For example, the healthcare industry has HIPPA regulations which stipulate what data is to be protected, how it is protected, and under what circumstances it can be disseminated. These are regulations put in place to protect the consumer, and stabilize competition between market providers.

A third area where IT represents a risk to a company’s competitive advantage is ineffective IT governance. According to Gartner (2013), “IT governance is defined as the processes that ensure the effective and efficient use of IT in enabling an organization to achieve its goals.” Throughout the past 30 years, companies struggled to define the role of IT as it related to business goals, many still do. IT was seen as a necessary evil, a means to an end, or another tool to automate certain tasks within a company, but not a way to achieve strategic advantage over a competitor or a method to dominate a market. As business goals evolve, formal IT governance will ensure that resource allocations remain dynamic and scalable to meet these changing needs.

A fourth risk area would be slow adoption, in that a company does respond to the challenges presented by direct competitors by updating or upgrading its technological capabilities. Many companies across all industries are slow to adopt new technologies, even if they offer clear advantages over current systems, due to user resistance to change or the excessive costs of redesigning proprietary applications to be compatible with modern systems. By not adopting new technologies, capabilities become limited, workers become unable to respond to customer demands in a timely manner, and systems can become overwhelmed to the point of system failure.

A final risk where IT represents a risk to a company’s competitive advantage is in cyber security. Any deficiency in a network’s security model presents a vulnerability that, if attacked with the correct vector, could represent a complete defacing of a company. The single most important aspect of any cyber security plan should be user education. There are a number of hardware and software solutions available to centralize and manage cyber security across an enterprise which provide comprehensive methods to thwart a direct attack from an outside entity, however they can only do so much. Half of all data breaches occur through phishing attacks, “in which unsuspecting users are tricked into downloading malware or handing over personal and business information” (IT Governance Ltd, 2015). These usually come in the form of a legitimate looking email and once the user initiates the connection, the system becomes infected and performs whatever it was programmed to do via the installed malware. The result of a breach could be catastrophic to an organization because of the importance of the actual data lost, and potentially the legal ramifications in the way of lawsuits from divulging protected data, whether inadvertent or on purpose.

IT Support of Competitive Advantage

A clear competitive advantage provided by IT is systems availability. With mission critical systems, redundancy is designed into the system model in an effort to eliminate the risk of system downtime and create 100% availability. While the expense of such a design can reduce net profits, it becomes a strategic advantage because a company is able to provide 24/7 services to their customers, regardless of geographic location. There are many companies moving their customer facing systems into cloud services to provide just that, availability. From online shopping, to financial institutions, to educational facilities, many companies have to provide a 24/7 model in order to meet customer demand and IT is the only way to ensure continuity and consistency across all communication methods.

IT provides a unique benefit for protecting sensitive and proprietary data in that the data can be encrypted to ensure only authorized users can gain access. Some regulations, such as HIPPA and PCI-DSS, stipulate not only data encryption but also low-level, whole drive encryption, using specific algorithms such as AES256 and a shared key pair. Encrypting data, and data communication channels, ensures that no outside party can view the information contained in these data files.

Proper implementation of IT governance can support a company’s competitive advantage because it ensures that all processes designed provide effective and efficient use of company resources using IT as the common thread. Over the past few years, as the value of IT proves its worth to companies looking to remain relevant in an ever changing consumer model, organizations have come to realize how important it is to bring IT goals in alignment with business goals. As an organization grows to meet market conditions, it becomes essential to align these two areas to ensure stability throughout the process. This provides the foundation necessary to ensure continuity as the company evolves.

A fourth way that IT supports a company’s competitive advantage is by enabling the company to be able to adapt quickly to changing markets. When implemented in an elastic model, such as the facilities provided with cloud solutions, companies can respond instantly to spikes in consumer demand with a few clicks of a mouse. By leveraging this model, companies can improve efficiencies, improve worker output, and lower operating costs, thereby increasing revenues and profits. A number of companies have adopted the agile model of development for their products, where concepts are quickly moved from the drawing board, to prototype, to final concept in a short time frame. Issues are fixed as they are found through use in a production environment. IT is the only way that this can be possible because of how the cloud model of scalability provides these resources in a dynamic way, as demanded.

A final way that IT supports an organization’s competitive advantage is through the implementation of a cohesive user education program and the implementation of an information security management system, which is a comprehensive approach to managing cyber security risks that takes into account not only people, but also processes, and technology. Security should be built into every process that any user takes to manipulate data in an information system. Once the physical perimeter of an infrastructure is also secured, users need to be trained to identify phishing attacks and social engineering tactics so they can become a weapon against these attack vectors rather than the weak link. Part of that training should include what cyber security systems are in place, how they protect users and corporate data, and why it is important for users to know this information.

IT Risk Scenario: System Availability

In the course of the author’s career, there was an instance where a major system outage resulted in the company losing a multi-million opportunity to a competitor. The root cause of the system outage was later found to be a misconfigured operating system update, provided by the software manufacturer as a critical update to patch a well-known vulnerability. This misconfigured system update caused every service hosted on the domain servers to reject every all queries from all systems. Since the update was automatically deployed to all servers in the forest, failover switching was not an option. It took over 6 hours to troubleshoot and eventually rebuild the primary server and supporting services to bring the network back online. In that time frame, a bid deadline expired for a major project and the author’s company was removed from consideration. Since they were one of only two companies that services this specific product group, from different factories, the contract was awarded to the competition. It represented a $20 million opportunity that spanned three years across five large developments. Had they been able to submit their bid, they would have saved the client 8% in costs, and over a month in lead-times.

IT Advantage Scenario: Data Privacy and Protection

Data security has become a major consideration for companies of all sizes, and for certain market segments it is a federal edict. Previous to the introduction if HIPPA regulations, the privacy of people’s health records were being mishandled. Data was stored in proprietary formats which increased administrative costs, and was shared with nearly anyone who had a seemingly legitimate need for it, whether that be for patient treatment or insurance carrier marketing purposes. Once public outcry reached critical mass, the Health Insurance Portability and Accountability (HIPPA) act of 1996 was created. HIPPA protects the confidentiality and security of healthcare information, and helps the healthcare industry control administrative costs (TN Department of Health, n.d.).

Conclusion

The implementation of IT systems comes with many risks and rewards for any entity, whether it be a company or a person. The main purpose of IT is to make a company more effective and efficient across all operational parameters. The proper management of the risks and advantages provided by an integrated IT platform can ensure that a business is able to meet the demand of its customers while being in a position to evolve as rapidly as their market does. Once systems and software are setup, security models implemented, and data secured, user education becomes the key component to ensuring that IT provides a secure platform for improved efficiencies and increased effectiveness expected across all job roles.





References
Garnter. (2013). IT Governance. Retrieved from http://www.gartner.com/it-glossary/it-governance

IT Governance Ltd. (2015). Federal IT professionals: insiders the greatest cybersecurity threat. Retrieved from http://www.itgovernanceusa.com/blog/federal-it-professionals-insiders-the-greatest-cybersecurity-threat/

TN Department of Health. (n.d.). HIPAA: Health Insurance Portability and Accountability Act. Retrieved from http://health.state.tn.us/hipaa/