#CrowdStrike Cause a Global Tech Outage - what happened, why, and (how) can it be prevented?

While the memes are amazingly good, and there's a lot of jest being spewed across the interwebs, this is a serious event with massive implications. So, in all seriousness, let's review the facts of the #CrowdStrike situation from 19-Jul-2024: 

As reported across global news outlets and the internets, a security company called CrowdStrike caused some chaos. There are cascading impacts across many industries. 

We are already seeing impacts: 

://courier service delays (UPS, FedEx, DHL, etc.) 

://flight delays/cancellations at the airport 

://small business closing for the day 

://websites being inaccessible 

://hospitals cancelling surgeries/treatments 

://municipalities being closed 

://government services being delayed 

among many other cascading effects that could last days, or weeks. 

While a major inconvenience, the bug was quickly resolved within CrowdStrike's system, so (as of publish date) the latest binaries are stable. Recovery will be slow and tedious, especially for larger networks, but the world will recover from this. 

What happened? As is being reported, a bug introduced during a routine update of their Falcon EDR software (anti-virus software run by millions and millions of customers) caused what is known as a kernel panic within the Windows operating system - we are seeing this manifest as a "bugcheck error" (aka - the Blue Screen Of Death , or #BSOD) on Windows machines. It does not affect #Apple or #Linux devices. Note: It is NOT a #Microsoft problem. 

How can we prevent this? Short answer, WE as users can't. However, this isn't the first time a large global tech vendor has caused major outages across the globe, and it won't be the last. 

How can CrowdStrike, or any another company, prevent this? Simply, adhering to the SDLC methodologies, adequate QA testing, and never do a full production roll out without fully testing in the field. A common practice is to deploy to 10% of the network and see how systems and users respond (yes sysadmins, you can do targeted deployments even if you don't have network segmentation in place). If all goes well, push to 25% and test again, then 50% and test again, then the full push. That way when a problem does occur, it doesn't take out everything and can be quickly fixed before a full production push. It's really IT Ops 101 - not that difficult. This is a good example of why you should backup your critical data frequently: whether to an external device, or a cloud storage facility (Google Drive, Dropbox, OneDrive, etc.). You should do this personally as often as you feel is necessary. Most companies have policies governing backup types, schedules, and testing methodologies. 

For my enterprise admins reading this, I hope you have a solid (and tested) backup methodology in place. Yes, you should test-restore your backups at least once per year, if not more often. If you can't restore the data, then what is the point of backing it up? 

So now the big question is, how does this issue get fixed? Well, it's a hands-on-machine fix (which means long days/nights and weekends for IT staffers for a bit). Since the devices are unable to boot, there's no back-of-house configuration that we admins can set to fix this. We literally have to put our hands on the device. The methodology is simple, and only takes about 5 minutes to do - but multiply that over hundreds, thousands, or even hundreds-of-thousands of devices and you can quickly see this is not a quick fix at scale. It is an even bigger nightmare for remote workers, who would need to be walked through the fix via telephone, making it a 30min fix (at best). In those cases, from my perspective, it makes more sense to send them a replacement machine that is not bricked, then reset the trouble device once back in hand. Hopefully you have the inventory ready and waiting, otherwise you need to grab a company credit card and hit up every electronics store in your city. What a fucking PITA. 

CrowdStrike's official guidance can be found on their webpage here: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ (external link). 

While all of this is happening, myself and most of my peers agree that CrowdStrike is still a quality vendor offering quality security products and services. This was just a BIG fuckup from whoever pushes out their updates. Clearly, someone did not follow protocol. 

As of this writing, CrowdStrike is the second largest security vendor in the world, which is why the impact of this was as massive as it was...and the cascade effect isn't done yet. There will be more fall out from this, not to mention the legal cases that could be brought against them in the aftermath due to the downtime. 

One of the biggest fallouts of this mess is phishing attacks - threat actors spinning up malicious domains claiming to fix the issue (they won't, they just want your money); emails being sent claiming to be able to fix the issue with "a click" (using a piggy-back technique to install a payload on your machine to do god knows what; oh and steal your money too). Please do not fall for the phish. It's won't end well for you, or your employer. 

There is no "easy button" here peeps. Just a massive Pain In The Ass. 

#StayCyberSecure 

#BeCyberAware

Security Systems Development Life Cycle (SecSDLC)

Security Systems Development Life Cycle

When designing information systems there are logical phases which must be considered in order to achieve maximum efficiency and effectiveness throughout the organization in every role. Throughout the six phases of the systems development life cycle (SDLC) it becomes imperative to ensure that security is integrated with each aspect of the platform. When building a security project, the same phases of the SDLC can be adapted to suite. The security systems development life cycle (SecSDLC) shares similarities with the SDLC, however the intent and activities are different. The purpose of this paper is to review and explain the phases of the SecSDLC, discussing the differences between the SDLC, and applicable certifications.

Investigation

In this phase, the project scope and goals are defined upper management. They provide the process methodologies, expected outcomes, project goals, the budget, and any other relevant constraints. “Frequently, this phase begins with an enterprise information security policy (EISP), which outlines the implementation of a security program within the organization.” (Whitman & Mattord, 2012, p. 26). Teams are organized, problems analyzed, and any additions to scope are defined, discussed, and integrated into the plan. The final stage is a feasibility study to determine if corporate resources are available to support the endeavor. The primary difference from the traditional SDLC is that management defines the project details. In the SDLC the business problems to be solved are researched and developed by the project team.

Analysis

In this phase, the documents gathers in phase one are studied and a preliminary analysis of the existing security polices is conducted. At the same time, the current threat landscape is evaluated and documented, as are the controls in place to manage or mitigate these threats. Included at this stage is a review of legal considerations that must be integrated into the security plan. The modern global threat landscape is such that any business, small or large, is susceptible to attack from a third party, whether it be directly or indirectly. Certain industries have strict requirements on how data is to be stored, shared, or manipulated. Standards such as HIPPA, NIST, PCI-DSS, the ISO27001 standard, and others provide guidelines for an organization to be certified as complaint with established processes and methods. Some industries require these certifications in order for a company to conduct business in that sector. Understanding state legislations with regards to what computer activities are deemed illegal is vital to the overall plan execution and sets the baseline for the types of security technologies that can be implemented across the enterprise. The risk assessment in this phase identifies, assesses, and evaluates the threats to the organization’s security and data. The final step in this phase is to document the findings and update the feasibility analysis. The main differences between the SDLC at this phase include the examination of legal issues, relevant standards based on the segment within which the company is situated, the completion of a formal risk analysis, and the review of the threat landscape and their underlying controls. Those aspects are specifically unique to the SecSDLC. While considering security within every phase of the SDLC is vital, the focus and scope of security considerations are vastly different compared to the SecSDLC which focuses solely on the security aspect of an information systems.

Logical Design

With the SecSDLC, this phase creates and develops the blueprints for information security across the enterprise. Key policies are examined and implemented, and an incident response plan is generated to ensure business continuity, define what steps are taken when an attack occurs, and what is done to recover from a disastrous event. Similar to the SDLC, applications, data support, and structures are selected considering multiple solutions in an approach to managing threats. Unique to the SecSDLC is the detail involved with securing the SDLC core concepts by analyzing the system security environment, functional security requirements, assurance that the security system developed will perform as expected, cost considerations with regards to hardware, software, personnel, and training, documentation of security controls that are planned or in place, security control development, use case tests and test evaluation methods. The concepts and best practices detailed by the NIST can be seen as a guide throughout this phase with regards to system hardening and expected security measures to be taken to ensure end-to-end security across the enterprise. Project documents are again updated, and as with previous phases, the feasibility study is revisited to determine whether or not to continue the project, and/or whether or not to outsource the project.

Physical Design

The fourth phase of the SecSDLC evaluates the information security technologies needed to support the created blueprint and generate alternative solutions, which dictate the final system design. Technologies evaluated in the logical design phase are the best are selected to support the solutions developed, whether they are custom built or off-the-shelf. A key component to this phase is developing a formal definition of what “success” means for the project implementation to be measured against. The design of physical security measures to support the proposed system are also included at this phase. Project documents are updated, refined, and a feasibility study is conducted to ensure the organization is prepared for system implementation. The final stage of this phase involves the presentation of the design to sponsors and stakeholders for review and final approval. If regulations such as HIPPA and/or PCI-DSS must be adhered to, the physical design the infrastructure components must be modeled after their specific requirements with regards to the machines data is stored on, how these machines are physically accessed, and how the data stored on these machines is disseminated to authorized parties. This is unique to the SecSDLC. While data access control is a standard consideration of any information system, HIPPA, for example, provides specific requirements in order to maintain the privacy of patient records and ensure that their data is only shared with specific authorized personnel within the medical industry. PCI-DSS covers how customer credit card details and identifiable data is stored, used, and accessed within a company’s network.

Implementation

This phase is similar to that of the SDLC. Selected solutions are purchased or developed, tested, implemented, and tested again. A penetration test could be conducted to ensure that the security measures installed perform as expected and the network resources are protected from third party intrusion. Personnel issues are revaluated, training and education programs conducted, and finally the complete package is presented to upper management for final sign off. The SDLC differs in this phase in that the system developed is rolled out to users for their daily use. The SecSDLC is implemented on the back end by network administrators, as approved by upper management. Aside from accessibility issues that are repaired during testing, the user has no involvement in this phase of the SecSDLC.

Maintenance and Change

This is the most important phase of the SecSDLC because of the evolving threat landscape. Older threats evolve and mature into more dangerous threats, and new threats aim for new attack vectors against system weaknesses. Active and constant monitoring, testing, modification, update, and repair must be conducted on information security systems in order to keep pace with maturing and emerging threats. Zero-day threats pose a significant threat to organizations at the cutting edge of their industry and their security plan must be flexible enough to be able to proactively prevent these threats while also integrating methods of recovery should an attack occur through an unknown vulnerability. This phase is the most different from the SDLC in that the SDLC framework is not designed to anticipate a software attack that requires a degree of application reconstruction. “In information security, the battle for stable, reliable systems is a defensive one” (Whitman & Mattord, 2012, p.29). The constant effort to repair damage and restore data against unseen attackers is a never ending process. Part of this phase includes the perpetual education of all personnel as new threats emerge and the security model is updated because an educated user is a powerful security tool.

Conclusion

The purpose of the SecSDLC is to provide the framework for designing and implementing a secure information system paradigm. Since it is based off the SDLC it shares many similarities in the processes and methods used to develop a comprehensive plan, but the intent and activities are different at each phase. While considering systems security is considered vital to every phase of the SDLC, the SecSDLC focuses solely on the implementation of technologies designed to protect an infrastructure from third party intrusion, data corruption, and data theft. The SDLC develops the systems used within a business, while the SecSDLC develops the system to protect these systems and an organization’s users.



ReferencesWhitman, M.E., & Mattord, H.J. (2012). Principles of Information Security (4th ed.). Retrieved from The University of Phoenix eBook Collection.

Market Segmentation - A Geek's Perspective

So one of the "Applications" sections of one of my chapter readings this week about market segmentation, coupled with a conversation I had earlier today with a great friend, spawned this blob of words.

Is the World Coming Closer Together? Many social commentators maintain that youth and teens are becoming more alike across countries over time. Others, although not disputing the fact, point out that differences between cultures at even younger ages by far exceed the similarities.

Take a position: People are becoming more and more similar, versus, The differences between people of different cultures far outweigh their similarities.

My reply:
Learning about segmentation was interesting this week. What it made me think about is how different everyone's tastes are, but also similar the world has become from where it has been. There was a question at the end of one of the chapters that asked that exactly. "Take a position: People are becoming more and more  similar versus The differences between people of different cultures far outweigh their similarities." I think our world is in a transitional period where the older generations are being moved out of their positions of power and influence, and the next generation is moving in with different worldly perspectives. At the moment, Baby Boomers are in positions of power around the world, but the Gen X and Y groups are quickly climbing the corporate and political ranks as the Boomers start to retire younger and younger. Age is becoming a non issue for the most part in a lot of industries as skill sets become a commodity that only experience can make more attractive, but consumers seem to be preferring interactions with younger workers, which demonstrates a level of segmentation from the consumer perspective as the younger and more modern generations start to become consumers of luxury items as their disposable income increases also at younger ages. I asked a group of teenagers (13-15 year olds) the same question. They pretty much all agreed with my theory. They look to interact with service providers that they can better relate to, and communicate at higher and higher levels of technical expertise on modern conveniences, rather than someone who reminds them of their parents, grandparents, or great-grandparents whom constantly offer their displeasure of such interactions. Racial lines are being blurred to the point of nonexistence in the younger generations waiting for their chance to be in positions of power and influence so they can make their mark in the world and change its perspective. Good marketers are being more creative with establishing brand messages that span most age groups and demographic groups around the planet, channel guidelines are well established and generally followed, and the gray area of taboo marketing has a tendency to go viral with the technically affluent community regardless of their age. The Internet offers a platform that supports extreme freedoms of expression, open sharing of information, and has a connectivity factor that has more than 4+ billion users around the globe a few milliseconds (or less) from each other. As such, marketing campaigns that would not be seen as appropriate (taboo) for conventional marketing channels like TV and radio are finding their place online where segmentation does not technically exist. With as connected as everyone is today, it makes sense why every interaction with our devices and machines renders some ad for a product that specifically appeals to you and your friends. Marketers are good at what they do, and technology makes it a lot easier to shift marketing methods at the speed of progress.

What do you think? Am I on the right track or completely off base? ~Geek

Reference:

Kotler, P., & Keller, K. L. (2012). Marketing Management (14th ed.). Retrieved from http://www.coursesmart.com/SR/7147203/9780132103008/617?__hdv=6.8.

What is an enterprise system is and how can this design support testing processes?

An enterprise system is a compilation of separate but related modulated wares that are integrated with a single cohesive database, with multiple interfaces, to achieve the business purpose of an organization across multiple departments, in an effort to consolidate separate legacy systems and improve overall efficiencies. Enterprise systems are complex by nature but with the right planning and proper execution success can be reached and the benefits to an organization can be immense.

The enterprise system paradigm supports testing processes by offering a wide range of potential test cases for every aspect of an organization, effectively enabling developers and systems engineers to better the system as a whole for the entire enterprise in specific ways. The concept behind enterprise systems is integration through modulation, empowering organizations to perform any function required and change the system on demand and/or based on local need. For example, an accounting module that is functionally sound for US locations will calculate salaries differently than what is required for a European location. Currencies are different, taxes are different, pay scales are different, etc. As such, modified/localized versions of modules allow the organization to deploy localized versions of the accounting module, while still integrating data to the central database. This allows executives from any locale to gain insight into labor trends and costs across the enterprise to make more intelligent decisions on the direction of the company on a global scale. Test cases can be created to compare modules and sub-modules to see which are transferable to other locations of the organizations, and then run in tandem to determine functionality. Since enterprise systems are sold by the module, then allow for some customization on the customers part, the accounting module in general should be transferable to any locale, with some minor modifications to allow for local laws and practices, which saves the company money overall. It is far easier and less expensive to modify an existing module to allow for proper payroll calculations based on local laws, as an example, than it is to have the developer write a completely new module for each location that requires it and then figure out how to integrate that data without having to add too much to the already complex central database. This testing model applies to any aspect of an enterprise system: inventory, human resources, manufacturing, etc, but with different data sets. Having a global infrastructure also allows administrators to tap into collective resources to evaluate and gain feedback on any proposed update/upgrade. Sometimes, asking workers simple questions can eliminate the need for many costly test cases, which when performed in excess can actually result in project failure due to never really gaining momentum and being stuck in procedure or policy, as it were. Sometimes just listening to the users can be an administrators best test case, as long as they are willing to hear what is said and then make sure that the executives buy into the concept.



Have a question? Have a comment? Comment below, let's start a dialog.






~Geek