- Delta Air Lines CEO Ed Bastian said the massive IT outage earlier this month that stranded thousands of customers will cost it $500 million.
- The airline canceled more than 4,000 flights in the wake of the outage, which was caused by a botched CrowdStrike software update and took thousands of Microsoft systems around the world offline.
- Bastian, speaking from Paris, told CNBC’s “Squawk Box” on Wednesday that the carrier would seek damages from the disruptions, adding, “We have no choice.”
Bastian said the figure includes not just lost revenue but “the tens of millions of dollars per day in compensation and hotels” over a period of five days. The amount is roughly in line with analysts’ estimates. Delta didn’t disclose how many customers were affected or how many canceled their flights.
It’s important to note that the DOT recently clarified a rule that reinforced that if an airline cancels a flight, they have to compensate the customer. So that’s the real reason why Delta had to spend so much, they couldn’t ignore their customers and had to pay out for their inconvenience.
So think about how much worse it might have been for fliers if a more industry-friendly Transportation Secretary were in charge. The airlines might not have had to pay out nearly as much to stranded customers, and we’d be hearing about how stranded fliers got nothing at all.
Why do news outlets keep calling it a Microsoft outage? It’s only a crowdstrike issue right? Microsoft doesn’t have anything to do with it?
It was a Crowdstrike-triggered issue that only affected Microsoft Windows machines. Crowdstrike on Linux didn’t have issues and Windows without Crowdstrike didn’t have issues. It’s appropriate to refer to it as a Microsoft-Crowdstrike outage.
Funny enough, crowdstrike on Linux had a very similar issue a few months back.
It’s similar. They did cause kernels to crash. But that’s because they hit and uncovered a bug in the ebpf sandboxing in the kernel, which has since been fixed
Are they actually shipping kernel modules? Why is this needed to protect from whatever it is they supposedly protect from?
I guess microsoft-crowdstrike is fair, since the OS doesn’t have any kind of protection against a shitty antivirus destroying it.
I keep seeing articles that just say “Microsoft outage”, even on major outlets like CNN.
To be clear, an operating system in an enterprise environment should have mechanisms to access and modify core system functions. Guard-railing anything that could cause an outage like this would make Microsoft a monopoly provider in any service category that requires this kind of access to work (antivirus, auditing, etc). That is arguably worse than incompetent IT departments hiring incompetent vendors to install malware across their fleets resulting in mass-downtime.
The key takeaway here isn’t that Microsoft should change windows to prevent this, it’s that Delta could have spent any number smaller than $500,000,000 on competent IT staffing and prevented this at a lower cost than letting it happen.
Its sort of like calling the terrorist attack on 911 the day the towers fell.
Although in my opinion, microsoft does have some blame here, but not for the individual outage, more for windows just being a shit system and for tricking people into relying on it.
Pretty sure their software’s legal agreement, and the corresponding enterprise legal agreement, already cover this.
The update was the first domino, but the real issue was the disarray of Delta’s IT Operations and their inability to adequately recover in a timely fashion. Sounds like a customer skimping on their lifecycle and capacity planning so that Ed can get just a bit bigger bonus for meeting his budget numbers.
Negligence can make contracts a little less permanent.
Delta was the only airline to suffer a long outage. That’s why I say Crowdstrike is the kickoff, but the poor, drawn-out response and time to resolve it is totally on Delta.
Idk, crowdstike had a few screwups in their pocket before this one. They might be on the hook for costs associated with an outage caused by negligence. I’m not a lawyer, but I do stand next to one in the elevator.
It breaks down once Delta begins arguing costs directly associated with their poor disaster recovery efforts.
Why is CrowdStrike responsible for Deltas poor practices?
Couldn’t agree more.
And now that this occurred, and cost $500m, perhaps finally some enterprise companies may actually resource IT departments better and allow them to do their work. But who am I kidding, that’s never going to happen if it hits bonuses and dividends :(
We just lost 500 million - we can’t afford that right now! /s
I wasn’t affected by this at all and only followed it on the news and through memes, but I thought this was something that needed hands-on-keyboard to fix, which I could see not being the fault of IT because they stopped planning for issues that couldn’t be handled remotely.
Was there some kind of automated way to fix all the machines remotely? Is there a way Delta could have gotten things working faster? I’m genuinely curious because this is one of those Windows things that I’m too Macintosh to understand.
There was no easy automated way if the systems were encrypted, which any sane organization mandates. So yes, did require hands-on-keyboard. But all the other airlines were up and running much faster, and they all had to perform the same fix.
Basically, in macOS terms, the OS fails to boot, so every system just goes to recovery only, and you need to manually enter the recovery lock and encryption password on every system to delete a file out of /System (which isn’t allowed in macOS because it’s read only but just go with it) before it will boot back into macOS. Hope you had those recorded/managed/backed up somewhere otherwise it’s a complete system reinstall…
So yeah, not fun for anyone involved.
Maybe not.
deleted by creator
Don’t worry everyone… Each and everyone of the CEOs involved in this debacle will earn millions this year and next and will eventually retire with more money they could possible spend in 10 lifetimes
If anything, they’ll continue to fall upwards completely deserving even more money
Additionally, don’t worry, they’ll just shift more costs onto the consumer and ultimately widen their profit-margins in no time.
Perhaps Boeing can save the airline industry a little more by lowering the costs of their planes by removing another bolt and jerry-rigging flight software onto an antiquated platform.
Good. They’ve been stealing from their customers for decades; this is fuckin’ karmic.
Also, maybe don’t put all your eggs into one single basket, from an infrastructure perspective.
Yeah, I say I as migrate another service to Azure…
That’s not just putting all your eggs into a single basket, that’s putting all your eggs into a rotting trashcan
Tell me you haven’t used Azure without telling me you haven’t used Azure.
Is Azure is fine. It is not amazing, it is not terrible, it is fine.
Our c-level team was super excited to announce we were migrating from AWS to Azure. “This is going to be so great for our infrastructure team!” The infrastructure team groaned. “But Azure is so much better!” Yeah, it’s fine. It’s all pros and cons. But migration sucks.
It’s like, we have sales people. We sell software. They know the software sales process. They know that sales is all pros, no cons. They know that the team that took them to dinner and golfing and gave them swag wouldn’t know an API from an APU. They know that migration is a major pain point. Why would they expect enthusiasm from the team that has to do it?
But they’re excited about it. It’s gonna be great.
No. The cost of lost profits will just be passed on to you.
I can’t wait to see crowdstrike get liquidated from all of this, MSOFT is getting so much flak when this straight up wasn’t their fault
Their stock is at +44% since July 2023, they might be fine
Pure gambling
Lawsuits haven’t started yet, too soon. Companies effected by the outage are still running number to see HOW effected they were
Why would they be liquidated?
Inability to pay the settlements on the inevitable lawsuits that will be coming their way for halting the world economy for a day
I’m sure their Terms of Service make it clear they have limited liability or need to go to arbitration.
Yea because that always holds up in court. I’m sure every legal team will claim the lack of QA was gross negligence on Crowdstrikes part and that normally allows vast portions of agreements to be nullified as one party clearly didn’t hold up their end of the deal
Crowdstrike wouldn’t have a business model if the security of Microsoft Windows wasn’t so awful. Microsoft isn’t directly to blame for this, but they’re not blameless either.
Windows defender for enterprise is a strong competitor in that market, and CISO that went with crowdstrike did it because the crowdstrike sales team hosts really great lunches and sponsors lots of sports teams
Sure, but they did send a $10 Uber Eats gift card, so you gotta take that into account.
499,999,990. Could be worse.
Edit: oh I see someone else beat me to it
Good thing they got that $10 Uber Eats card.
Aw that’s a shame. Poor rich company.
Lol
Crowdstrike offers layered rollouts, but some executive declined this because they want the most up to date software at all times.
But at least a door didn’t just fall off a plane. That’s already fixed in windows 11 the shitwagon release.
Yeah… Maybe don’t put all your IT eggs in one basket next time.
Delta is the one that chose to use Crowdstrike on so many critical systems therefore the fault still lies with Delta.
Every big company thinks that when they outsource a solution or buy software they’re getting out of some responsibility. They’re not. When that 3rd party causes a critical failure the proverbial finger still points at the company that chose to use the 3rd party.
The shareholders of Delta should hold this guy responsible for this failure. They shouldn’t let him get away with blaming Crowdstrike.
So you think Delta should’ve had a different antivirus/EDR running on every computer?
I think what @[email protected] was saying is you shouldn’t have multiple mission critical systems all using the same 3rd party services. Have a mix of at least two, so if one 3rd party service goes down not everything goes down with it
That sounds easy to say, but in execution it would be massively complicated. Modern enterprises are littered with 3rd party services all over the place. The alternative is writing and maintaining your own solution in house, which is an incredibly heavy lift to cover the entirety of all services needed in the enterprise. Most large enterprises are resources starved as is, and this suggestion of having redundancy for any 3rd party service that touches mission critical workloads would probably increase burden and costs by at least 50%. I don’t see that happening in commercial companies.
As far as the companies go, their lack of resources is an entirely self-inflicted problem, because they’re won’t invest in increasing those resources, like more IT infrastructure and staff. It’s the same as many companies that keep terrible backups of their data (if any) when they’re not bound to by the law, because they simply don’t want to pay for it, even though it could very well save them from ruin.
The crowdstrike incident was as bad as it was exactly because loads of companies had their eggs in one basket. Those that didn’t recovered much quicker. Redundancy is the lesson to take from this that none of them will learn.
As far as the companies go, their lack of resources is an entirely self-inflicted problem, because they’re won’t invest in increasing those resources, like more IT infrastructure and staff.
Play that out to its logical conclusion.
- Our example airline suddenly doubles or triples its IT budget.
- The increased costs don’t actually increase profit it merely increases resiliency
- Other airlines don’t do this.
- Our example airline has to increase ticket prices or fees to cover the increased IT spending.
- Other airlines don’t do this.
- Customers start predominantly flying the other airlines with their cheaper fares.
- Our example airline goes out of business, or gets acquired by one of the other airlines
The end result is all operating airlines are back to the prior stance.
customers start predominantly flying the other airlines with cheaper fares
I was with you till this part, except with the way flying is set up in this country, there’s very little competition between airlines. They’ve essentially set themselves up with airports/hubs so if an airline is down for a day, that’s kinda it unless you want to switch to a different airport.
CNBC Media Bias Fact Check Credibility: [High] (Click to view Full Report)
CNBC is rated with High Creditability by Media Bias Fact Check.
Bias: Left-Center
Factual Reporting: Mostly Factual
Country: United States of America
Full Report: https://mediabiasfactcheck.com/cnbc/Check the bias and credibility of this article on Ground.News
Thanks to Media Bias Fact Check for their access to the API.
Please consider supporting them by donating.Footer
Media Bias Fact Check is a fact-checking website that rates the bias and credibility of news sources. They are known for their comprehensive and detailed reports.
Beep boop. This action was performed automatically. If you dont like me then please block me.💔
If you have any questions or comments about me, you can make a post to LW Support lemmy community.
