Microsoft says it estimates that 8.5m computers around the world were disabled by the global IT outage.
It’s the first time a figure has been put on the incident and suggests it could be the worst cyber event in history.
The glitch came from a security company called CrowdStrike which sent out a corrupted software update to its huge number of customers.
Microsoft, which is helping customers recover said in a blog post: “We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices.”
I wonder how much this cost people & businesses.
For instance, people’s flights were canceled because of this resulting in them having to stay in hotels overnight. I’m sure there’s many other examples.
For businesses, a lot of them are hiring IT companies (consultants, MSPs, VARs, and whoever the hell else they can get) at a couple to a few hundred bucks an hour per person to get boots on the ground to fix it. Some of them have everyone below the C levels with any sort of technical background doing entry level work so there’s also lost opportunity cost.
I was in that industry for a long time and still have a lot of colleagues there. There’s a guy I know making almost $200k/yr out there at desks trying to help fix it. He moved into an SRE role years ago so that’s languishing this week while he’s going desk to desk and office to office with support staff and IT contractors.
At least two large companies have an API where they’re paying for a pile of compute and currently have a small fraction of use. Companies are paying to use those APIs but can’t.
I don’t know if there’s a good way to actually figure out how much this is costing because there are so many variables. But you can bet there are a few people at the top funneling that money directly to themselves, never to be seen again.
That’s kind of what I thinking. There’s countless ways this costs money. And not an insignificant amount either.
Also, I work IT and have been in vacation. So sad I am missing all this!
Something I didn’t think about but has since come to my attention (group chat is getting spicy) is that there are a lot of mid level IT folks on salary who are getting the absolute dog shit worked out of them right now without seeing an extra dime. So the costs are beyond monetary.
8.5M worldwide? I was expecting higher numbers, interesting
Even if 8.5m is correct, with many being servers, the total people affected is much much higher.
The downstream effects are likely much much greater. If an auth server/DB server/API server/etc (for example) got taken down, the failure cascades
The idea that any such servers would be running windows… shudder
In the corpo that I work in, we had about 3000 servers down, plus probably twice as many workstations including laptops of remote workers. Yeah, fun!
There are a lot of misunderstandings about what happened. First, the ‘update’ was to a data file used by the crowdstrike kernel components (specifically ‘falcon’.) while this file has a ‘.sys’ name, it is not a driver, it provides threat definition data. It is read by the falcon driver(s), not loaded as an executable.
Microsoft doesn’t update this file, crowdstrike user mode services do that, and they do that very frequently as part of their real-time threat detection and mitigation.
The updates are essential. There is no opportunity for IT to manage or test these updates other than blocking them via external firewalls.
The falcon kernel components apparently do not protect against a corrupted data file, or the corruption in this case evaded that protection. This is such an obvious vulnerability that i am leaning toward a deliberate manipulation of the data file to exploit a discovered vulnerability in their handling of a malformed data file. I have no evidence for that other than resilience against malformed data input is very basic software engineering and crowdstrike is a very sophisticated system.
I’m more interested in how the file got corrupted before distribution.
Yeah, how the hell did this failure pass testing, is what I want to know!
It’s the cyber 9/11 they always worried about.
Y2K, delayed 24 years, 7 months, and 19 days.
What worries me even more is that something pretty similar could happen to 32-bit devices in 2038.
In case you needed to another reason to switch to Linux.
Windows is so unreliable that even Microsoft runs Linux internally.
When this happened to Linux and MacOS users of Crowdstrike some time ago, no one cared.
CrowdStrike will ultimately have contract terms that put responsibility on the companies, and truth be told the companies should be able to handle this situation with relative ease. Maybe the discussion here should be on the fragility of Windows and why Linux is a better option.
In this case, it’s really not a Linux/windows thing except by the most tenuous reasoning.
A corrupted piece of kernel level software is going to cause issues in any OS.
Cloudstrike itself has actually caused kernel panics on Linux before, albeit less because of a corrupted driver and more because of programming choices interacting with kernel behavior. (Two bugs: you shouldn’t have done that, and it shouldn’t have let you).Tenuously, Linux is a better choice because it doesn’t need this type of software as much. It’s easier and more efficient to do packet inspection via dedicated firewall for infrastructure, and the other parts are already handled by automation and reporting tools you already use.
You still need something in this category if you need to solve the exact problem of “realtime network and filesystem event monitoring on each host”, but Linux makes it easier to get right up to that point without diving into the kernel.
Also vendors managing auto update is just less of a thing on Linux, so it’s more the cultural norm to manage updates in a way that’s conducive to staggering that would have caught this.Contract wise, I’m less confident that crowd strike has favorable terms.
It’s usually consumers who are straddled with atrocious terms because they neither have power nor the interest in digging into the specifics too far.
Businesses, particularly ones that need or are interested in this category of software, inevitably have lawyers to go over contract terms in much more detail and much more ability to refuse terms and have it matter to the vendor. United airlines isn’t going to accept the contract terms of caveat emptor.You assume that businesses operate in good faith. That they thoroughly review contracts to ensure that they are fair and in the best interests of all its employees. Do you really think Greg, a VP of Cloud Solutions that makes 500k a year, who gets his IT advice on the golf course by AWS, Microsoft, & Oracle reps. Who gets wined & dined almost weekly by these reps, and a speaking spot at re:Invent, and believes Gartner when it says spending $5 million a month on cloud hosting and $90/TB on Egress traffic is normal, has the company’s best interests in mind?
I’ve seen companies pay millions for things they never used, or that weren’t ever provided by the vendor. You go to your managers, and say… “hey, why are we paying for this?” and suddenly you’re the bad guy. I’d love for you to prove me wrong. I’ve found pieces of progress before, within isolated teams when a manager wanted to actually accomplish something. It never lasts though… its like being an ice cube in a glass full of warm water.
There’s a big difference between “buying stuff you don’t need”, and “not having legal review a contract”, or “accepting terms that include no liability”.
Buying stuff you don’t need is in the authority of a VP seeing as their job is to make choices. Bypassing legal review and accounting diligence controls typically isn’t at any company big enough to matter.
I trust your hypothetical VP to not want to get fired from his nice job by skipping the paperwork for a done deal.Do you honestly think that Amazon just didn’t read the contract? Microsoft? Google? The US government?
They’re getting sued, and they’re gonna have to pay some money. Cynicism is one thing, but taking it to the degree of believing that people are signing unread contracts that waive liability for direct, attributable damage caused by unprofessional negligence is just assinine.
