All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

  • jedibob5@lemmy.world
    link
    fedilink
    English
    arrow-up
    158
    arrow-down
    3
    ·
    6 months ago

    Reading into the updates some more… I’m starting to think this might just destroy CloudStrike as a company altogether. Between the mountain of lawsuits almost certainly incoming and the total destruction of any public trust in the company, I don’t see how they survive this. Just absolutely catastrophic on all fronts.

    • RegalPotoo@lemmy.world
      link
      fedilink
      English
      arrow-up
      38
      arrow-down
      2
      ·
      6 months ago

      Agreed, this will probably kill them over the next few years unless they can really magic up something.

      They probably don’t get sued - their contracts will have indemnity clauses against exactly this kind of thing, so unless they seriously misrepresented what their product does, this probably isn’t a contract breach.

      If you are running crowdstrike, it’s probably because you have some regulatory obligations and an auditor to appease - you aren’t going to be able to just turn it off overnight, but I’m sure there are going to be some pretty awkward meetings when it comes to contract renewals in the next year, and I can’t imagine them seeing much growth

      • jedibob5@lemmy.world
        link
        fedilink
        English
        arrow-up
        5
        ·
        edit-2
        6 months ago

        Don’t most indemnity clauses have exceptions for gross negligence? Pushing out an update this destructive without it getting caught by any quality control checks sure seems grossly negligent.

    • ThrowawaySobriquet@lemmy.world
      link
      fedilink
      English
      arrow-up
      18
      ·
      6 months ago

      I think you’re on the nose, here. I laughed at the headline, but the more I read the more I see how fucked they are. Airlines. Industrial plants. Fucking governments. This one is big in a way that will likely get used as a case study.

    • Bell@lemmy.world
      link
      fedilink
      English
      arrow-up
      5
      arrow-down
      6
      ·
      6 months ago

      Don’t we blame MS at least as much? How does MS let an update like this push through their Windows Update system? How does an application update make the whole OS unable to boot? Blue screens on Windows have been around for decades, why don’t we have a better recovery system?

      • sandalbucket@lemmy.world
        link
        fedilink
        English
        arrow-up
        9
        arrow-down
        1
        ·
        6 months ago

        Crowdstrike runs at ring 0, effectively as part of the kernel. Like a device driver. There are no safeguards at that level. Extreme testing and diligence is required, because these are the consequences for getting it wrong. This is entirely on crowdstrike.

  • Encrypt-Keeper@lemmy.world
    link
    fedilink
    English
    arrow-up
    86
    ·
    6 months ago

    Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao

  • boaratio@lemmy.world
    link
    fedilink
    English
    arrow-up
    71
    ·
    6 months ago

    CrowdStrike: It’s Friday, let’s throw it over the wall to production. See you all on Monday!

  • richtellyard@lemmy.world
    link
    fedilink
    English
    arrow-up
    64
    ·
    6 months ago

    This is going to be a Big Deal for a whole lot of people. I don’t know all the companies and industries that use Crowdstrike but I might guess it will result in airline delays, banking outages, and hospital computer systems failing. Hopefully nobody gets hurt because of it.

    • RegalPotoo@lemmy.world
      link
      fedilink
      English
      arrow-up
      24
      ·
      6 months ago

      Big chunk of New Zealands banks apparently run it, cos 3 of the big ones can’t do credit card transactions right now

      • index@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        3
        arrow-down
        10
        ·
        6 months ago

        cos 3 of the big ones can’t do credit card transactions right now

        Bitcoin still up and running perhaps people can use that

        • I_Miss_Daniel@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          6 months ago

          Bitcoin Cash maybe. Didn’t they bork Bitcoin (Core) so you have to wait for confirmations in the next block?

  • Sʏʟᴇɴᴄᴇ@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    41
    ·
    6 months ago

    Yep, stuck at the airport currently. All flights grounded. All major grocery store chains and banks also impacted. Bad day to be a crowdstrike employee!

    • iknowitwheniseeit@lemmynsfw.com
      link
      fedilink
      English
      arrow-up
      6
      arrow-down
      1
      ·
      6 months ago

      My flight was canceled. Luckily that was a partner airline. My actual airline rebooked me on a direct flight. Leaves 3 hours later and arrives earlier. Lower carbon footprint. So, except that I’m standing in queue so someone can inspect my documents it’s basically a win for me. 😆

  • ari_verse@lemm.ee
    link
    fedilink
    English
    arrow-up
    38
    arrow-down
    2
    ·
    6 months ago

    A few years ago when my org got the ask to deploy the CS agent in linux production servers and I also saw it getting deployed in thousands of windows and mac desktops all across, the first thought that came to mind was “massive single point of failure and security threat”, as we were putting all the trust in a single relatively small company that will (has?) become the favorite target of all the bad actors across the planet. How long before it gets into trouble, either because if it’s own doing or due to others?

    I guess that we now know

  • scripthook@lemmy.world
    link
    fedilink
    English
    arrow-up
    35
    ·
    6 months ago

    crowdstrike sent a corrupt file with a software update for windows servers. this caused a blue screen of death on all the windows servers globally for crowdstrike clients causing that blue screen of death. even people in my company. luckily i shut off my computer at the end of the day and missed the update. It’s not an OTA fix. they have to go into every data center and manually fix all the computer servers. some of these severs have encryption. I see a very big lawsuit coming…

    • MataVatnik@lemmy.world
      link
      fedilink
      English
      arrow-up
      8
      arrow-down
      1
      ·
      edit-2
      6 months ago

      they have to go into every data center and manually fix all the computer servers.

      Jesus christ, you would think that (a) the company would have safeguards in place and (b) businesses using the product would do better due diligence. Goes to show thwre are no grown ups in the room inside these massive corporations that rule every aspect of our lives.

      I’m calling it now. In the future there will be some software update for your electric car, and due to some jackass, millions of cars will end up getting bricked in the middle of the road where they have to manually be rebooted.

      • yesoutwater@lemm.ee
        link
        fedilink
        English
        arrow-up
        3
        arrow-down
        1
        ·
        6 months ago

        I work for one of these behemoths, and there are a lot of adults in the room. When we began our transition off the prior, well known corporate AV, I never even heard of crowd strike.

        The adults were asking reasonable questions: why such an aggressive migration timeline? Why can’t we have our vendor recommended exclusion lists applied? Why does this need to be installed here when previously agentless technologies was sufficient? Why is crowd strike spending monies on a Superbowl ad instead of investing back into the technology?

        Either something fucky is a foot, as in this was mandated to our higher ups to m make the switch (why?), or, as is typically the case, the decision was made already and this ‘due diligence’ is all window dressing to CYA.

        Who gives a shit about fines on SLAs if your vendor is going to foot the bill.

  • BurnSquirrel@lemmy.world
    link
    fedilink
    English
    arrow-up
    50
    arrow-down
    16
    ·
    6 months ago

    I’m so exhausted… This is madness. As a Linux user I’ve busy all day telling people with bricked PCs that Linux is better but there are just so many. It never ends. I think this is outage is going to keep me busy all weekend.

  • ililiililiililiilili@lemm.ee
    link
    fedilink
    English
    arrow-up
    33
    ·
    6 months ago

    My dad needed a CT scan this evening and the local ER’s system for reading the images was down. So they sent him via ambulance to a different hospital 40 miles away. Now I’m reading tonight that CrowdStrike may be to blame.

  • StaySquared@lemmy.world
    link
    fedilink
    English
    arrow-up
    33
    ·
    edit-2
    6 months ago

    Been at work since 5AM… finally finished deleting the C-00000291*.sys file in CrowdStrike directory.

    182 machines total. Thankfully the process in of itself takes about 2-3 minutes. For virtual machines, it’s a bit of a pain, at least in this org.

    lmao I feel kinda bad for those companies that have 10k+ endpoints to do this to. Eff… that. Lot’s of immediate short term contract hires for that, I imagine.

      • StaySquared@lemmy.world
        link
        fedilink
        English
        arrow-up
        4
        ·
        edit-2
        6 months ago

        That’s one of those situations where they need to immediately hire local contractors to those remote sites. This outage literally requires touching the equipment. lol

        I’d even say, fly out each individual team member to those sites… but even the airports are down.

      • catloaf@lemm.ee
        link
        fedilink
        English
        arrow-up
        4
        ·
        6 months ago

        Call the remote people in, deputize anyone who can work a command line, and prioritize the important stuff.

    • uis@lemm.ee
      link
      fedilink
      English
      arrow-up
      1
      ·
      6 months ago

      Lot’s of immediate short term contract hires for that, I imagine.

      I think sysadmins union should be created today

    • qjkxbmwvz@startrek.website
      link
      fedilink
      English
      arrow-up
      1
      ·
      6 months ago

      Can you program some keyboard-presenting device to automate this? Still requires plugging in something of course…what a mess.

      • trolololol@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        6 months ago

        Yep I have one of these, I think it’s called tiny. Very similar to an Arduino, and very easy to program.

  • misterkiem@lemmy.world
    link
    fedilink
    English
    arrow-up
    36
    arrow-down
    3
    ·
    edit-2
    6 months ago

    lol

    too bad me posting this will bump the comment count though. maybe we should try to keep the vote count to 404

    • a Kendrick fan@lemmy.ml
      link
      fedilink
      English
      arrow-up
      4
      arrow-down
      4
      ·
      6 months ago

      I can only see 368 comments rn, there must be some weird-ass puritan server blocking .ml users. It’s not beehaw as I can see comments from there.

      I can only conclude that it is probably some liberals trying to block “Tankies” and no comment of value was lost.