CrowdStrike fixes start at “reboot up to 15 times” and get more complex from there
- Thread starter JournalBot
- Start date
I'm wondering how CrowdStrike as a company fares from this issue? Stock is currently down 10%
Upvote
152
(154
/
-2)
This underscores what a terrifying responsibility it is to push out updates. I'm basically shaking when we push out updates to our product, especially because iOS/Android deployments are essentially impossible to debug. At least on desktop, we can get people to go delete a file. We can't even do that on mobile. We rely on a witches brew of safe modes.
I can't tell if CrowdStrike were sloppy in their testing. But in all likelihood, they just tested on systems that were a little too perfectly configured, and when it hit the real world, it exploded. And maybe their rollout wasn't tiered enough.
My sympathies. Having your code be a core driver on many of the world's systems is as awesome as that responsibility can get.
I can't tell if CrowdStrike were sloppy in their testing. But in all likelihood, they just tested on systems that were a little too perfectly configured, and when it hit the real world, it exploded. And maybe their rollout wasn't tiered enough.
My sympathies. Having your code be a core driver on many of the world's systems is as awesome as that responsibility can get.
Upvote
430
(444
/
-14)
Upvote
377
(387
/
-10)
It really feels like even just… a moderately passable attempt at best practices with update deployments could avoid all of this? There’s no way all these major enterprises are installing software that can just update itself without telling anybody, right?
Which would mean that somebody, somewhere at most of these companies pushed play on the update without installing it on so much is a single test box first?
Either that, or it was more of a Trojan thing where the issue didn’t start popping up for a while after the update?
Which would mean that somebody, somewhere at most of these companies pushed play on the update without installing it on so much is a single test box first?
Either that, or it was more of a Trojan thing where the issue didn’t start popping up for a while after the update?
Upvote
30
(74
/
-44)
I dunno, they seem to be living up to their name here already; they struck the crowd, alright, open-handed right across the boot process.
Upvote
147
(151
/
-4)
I'm confused how seeing their name in the stack trace means that the rest of that story has to be true ...
Upvote
235
(243
/
-8)
At about 3:30pm my admins got me to bypass bitlocker, log on as an admin in safe mode and rename the file. Oddly explorer failed every time I clicked to rename - so I ended up using CMD.exe.
Took about 20 minutes as they knew my bitlocker bypass key. This is a solution that scales horribly.
Funnily enough, today was the best weather of the year so far. Terrible shame for an end user like me. Terrible, terrible shame
Took about 20 minutes as they knew my bitlocker bypass key. This is a solution that scales horribly.
Funnily enough, today was the best weather of the year so far. Terrible shame for an end user like me. Terrible, terrible shame
Upvote
301
(304
/
-3)
I don't think it's a formatting issue, as the dash in the filename is missing on desktop as well, and as far as I can tell, also in the source.
The "<code>" formatting for the filename is causing a line break, but that's less of an issue than the filename being incorrect.
Upvote
29
(29
/
0)
Agile ≠ no QA. CrowdStrike's is just a shitty development process it seems.
Upvote
170
(183
/
-13)
Have you tried turning it off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on again?
Upvote
744
(747
/
-3)
With great power comes great responsibility, and it seems they didn't take this responsibility seriously enough. Especially since it had managed to come into critical applications like hospitals, this is insane. Microsoft etc at least staggers update somewhat.
Upvote
117
(117
/
0)
Upvote
474
(485
/
-11)
Upvote
327
(332
/
-5)
I suspect their customers and rival vendors might already refer to it as being Crowdstruck.
Upvote
62
(62
/
0)
Because if it doesn't, it sucks at being AV software.
It's the give and take of risk of all AV software.
Granted, still shouldn't have happened.
Upvote
162
(168
/
-6)
When it's Thursday in the USA it's Friday in Australia and Japan. The answer isn't to not deploy, it's thorough testing and a progressive rollout.
Upvote
154
(159
/
-5)
Upvote
143
(143
/
0)
"Hello, IT. Have you tried turning it off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on and off and on again?"
Edit: Oops, missed the original several posts up. I'm glad we're all on the same page though.
Edit: Oops, missed the original several posts up. I'm glad we're all on the same page though.
Upvote
198
(206
/
-8)
Indeed. I've been burned in many much more minor ways by updates and am now extremely reluctant to install any. That's probably too conservative, but where is the sane middle ground??
One thing is for sure: all your eggs in one basket saves on basket costs, but you pay for it dearly at some later point.
Upvote
64
(65
/
-1)
Is it just me, or did anyone else read this and flash back to the time when their dad kept banging on the side of the old CRT TV's to get it to show a picture?
Upvote
106
(107
/
-1)
Upvote
86
(86
/
0)
Best practices fail…for the win!
Too cheap, too lazy, too understaffed, too undertrained; take your pick or the combo meal.
I built a website for a local NPO last year, and wrote them a detailed change control guide and why SSL was not an option, and so on. They have some smart people, including several IT pros. I figured “they got it”. I did wonder if webmaster by committee was a blessing or a curse.
Their entire membership list and passwords were published online last week.
Curse it is.
Upvote
93
(96
/
-3)
If Hector Martin's analysis is correct, Crowdstrike do file parsing in kernel space, and the driver shat itself on a malformed update file. I thought Tavis Ormandy shamed security companies into not doing stupid shit like that years ago.
Upvote
201
(202
/
-1)
"If rebooting multiple times isn't fixing your problem, Microsoft recommends..."
checking that you are plugged in.
checking that you are plugged in.
Upvote
36
(40
/
-4)
I feel for all those poor souls responsible for fixing this on their network, many of whom will be personally blamed by non-technical higher ups (at least at first). I've been there.
Upvote
167
(167
/
0)