What we know about the xz Utils backdoor that almost infected the world

Malicious updates made to a ubiquitous tool were a few weeks away from going mainstream.

See full article...

 

[SeanJW]

But picking another timezone doesn’t actually hide where you live. You are most likely to have a pattern of times for commits that will point to your real timezone.

In general I'd agree with this - but open source committers can be doing it on the clock as a part of their job, after hours outside their job, or looking at the commit history on my personal crap as an example, "complete chaos".

 

[havokcla5h]

So this is funny as around the time this article was written was april fools right. It gets more entertaining as someone spot on said this would make a good silicon valley episode. Then you add in that the guy who found it was named Andres Freund... A. Friend. Come on!

Me I think the real conspiracy is that this happened at all. And if it did they planted it. All this is missing is Michael Douglas and Demi Moore, who knows maybe they will be in the movie about this or guest stars on the silicon valley episode.

That said ya any commits to ANYTHING should be thoroughly reviewed and tested. I know people are like Lemmings and install and DL anything from github and docker, but come on lets save those Darwin awards for those who really need them.

 

[MalEbenSo]

"Working" is the important word here. As far as I can tell from reading the news, a big reason this exploit almost succeeded was because no one was getting paid to work on xz. "So pay 'em!" you say. Where is the money gonna come from?

Indeed.

I know how to donate to Wikipedia, Signal, the Internet Archive, …

But is there a similar foundation that will distribute donated funds among maintainers?

 

[MalEbenSo]

Performance hit is quite substantial, actually. I have no doubt that this thing would have been detected, eventually. However, it might have happened months from now. Then it would have been everywhere already.

But this is a good thing. A very good thing, actually.

There have been discussions about supply chain attacks, for years. Decades, actually. We used to call it "poisoning the well" many years ago. But no matter how much we talk about it, it was all theoretical. I mean, people even assumed that compilers have been backdoored many years ago. But noone was going to spend this much effort just to show that it was possible and to make people accept the possibility. So not much was really done about it.

Until now.

Now we are already seeing changes being made to OpenSSH that would have not been possible few months ago. Native systemd notification integration is already been developed (since 30th of March), so no need for libsystemd linking anymore. It will take some time to get integrated but it will happen. We are seeing people understanding that there is absolutely no need to have binary blobs in source repositories (except rare cases, of course, but those are going to be audited even more now). Checking source repositories against tarballs have been done before, many times. But obviously it wasn't good enough or often enough. That will change as well. People being dicks to maintainers are going to get greeted with "go fuck yourself" now, without a second thought. It will be extreme but it will be safer. For eternity I was terrified of compiling software myself because every time I invoked "./configure ..." I would think "fuck knows what is going on there right now". I did occasionally check scripts, I would grep for unexpected things but I was aware I'd never detect a very skilled attacker, like this one. Now there is going to be much more checking of autoconf/make/CMake/etc files in source repos. It won't be easy to detect things, but it will be easier. More eyes will be put on sources. For example, I am going to pick a random smaller project and just read the commit history, look for oddities, etc. Not because I expect to find something but I want to see what else should be looked at, etc. Eventually, I might end up with toolset that might help speed this process up. So there will be at least one more set of eyes looking at sources. I imagine that companies/organizations with more resources are going to put tons of effort into automating all this. So yeah, xz backdoor is actually a good thing, in a very bizarre way.

Also, I can't hunt all the references at the moment but I believe it was certificate (not the SSH key) that is used as a vector of attack, because certs are checked early and no configuration options will disable that check, while it wouldn't be the case with keys. A change to OpenSSH has already been suggested so OpenSSH will only get more secure because of this and one less vector of attack is now available.

Amount of skill and time/effort invested in this is mind blowing. I don't think people outside security really comprehend the skill/time involved here, this was insanely well executed attack. My first thought was "This had to be TURLA" because it was insanely smart and whoever did this had lots of patience. This does not (and will not) happen often.

So yeah, we were incredibly lucky that a Postgres developer caught it early.

However, it is mind blowing how many times security incidents have been detected by looking at CPU/RAM usage on systems, it is really no surprise that this is how xz backdoor got detected.

Would (or should) intrusion detection systems or monitored firewalls have detected this?

Ideally before it even got out?

Like: Where is this connection comming from? Why is it using CPU cycles in this untypical manner?

I have no idea what is possible these days with fixed rules, heuristics and AI.

 

[HobartTas]

I realize this is Linux specific but being a Windows user I recall the last time I compressed something using Winrar or Winzip was probably about 5-10 years ago, and even then I used the compression=none option because basically I just wanted to send a large file about 1.5 GB in size to someone over the internet and just wanted to generate a checksum to make sure they got it correctly. There was no point in compressing it at all given today's internet speeds and also with hard drives and SSD's being in the multi-TB range then again there's no point either.

I think I used those utilities regularly before the turn of the century and when Y2K clicked over then usage has basically tanked. There's also the manual labor in having to use it and spending a few minutes crafting all the necessary options and switches you might need to the basic command which again means since I don't use it then there's a lot of time saved.

The only time I actually use compression is to set it to being on globally when using ZFS and creating a new pool using the default setting. Is this XZ utility something that is automatically invoked when for example downloading Linux software or updates to decompress it? If I was using Linux I can't really see why I'd ever use this either except in the circumstance I detailed above.

 

[numerobis]

I realize this is Linux specific but being a Windows user I recall the last time I compressed something using Winrar or Winzip was probably about 5-10 years ago, and even then I used the compression=none option because basically I just wanted to send a large file about 1.5 GB in size to someone over the internet and just wanted to generate a checksum to make sure they got it correctly. There was no point in compressing it at all given today's internet speeds and also with hard drives and SSD's being in the multi-TB range then again there's no point either.

I think I used those utilities regularly before the turn of the century and when Y2K clicked over then usage has basically tanked. There's also the manual labor in having to use it and spending a few minutes crafting all the necessary options and switches you might need to the basic command which again means since I don't use it then there's a lot of time saved.

The only time I actually use compression is to set it to being on globally when using ZFS and creating a new pool using the default setting. Is this XZ utility something that is automatically invoked when for example downloading Linux software or updates to decompress it? If I was using Linux I can't really see why I'd ever use this either except in the circumstance I detailed above.

You're not using the utility manually but your browsing and software update downloads are all compressed and decompressed for you.

 

[SeanJW]

You're not using the utility manually but your browsing and software update downloads are all compressed and decompressed for you.

Heh. And there's follow-on effects even for unrelated stuff. The Go xz module (which is completely unrelated, a Go native code developed and maintained by some poor sod who definitely doesn't get enough credit for it) has released a new version. Not because it was affected but to include a notice in the security docs saying "not me! really not me! not even close to me!"

(autodependency checks picked up the version change and I poked to see what changed seeing the timing was odd and I didn't expect it to be affected)

 

[malor]

The only time I actually use compression is to set it to being on globally when using ZFS and creating a new pool using the default setting. Is this XZ utility something that is automatically invoked when for example downloading Linux software or updates to decompress it? If I was using Linux I can't really see why I'd ever use this either except in the circumstance I detailed above.

You're talking about two different things. ZFS compression is built into the filesystem. If you have compression enabled for a volume (generally 'zstd' is the best current option), then any file you store on the volume will automatically be compressed. It's decompressed when it's read back, so that you can't even see the compressed bytes on disk. The file works like normal, but it takes up less space. (and usually runs at the same speed or faster; CPU compression/decompression is generally much faster than disks, even SSDs. zstd is very efficient.) The whole compression/decompression is internal to ZFS, and invisible to you.

The xzutils, on the other hand, are not at all transparent. They're command line utilities that take either files or bytestreams and produce compressed files or bytestreams. I most frequently use them by passing arguments to the 'tar' utility, eg: tar -cJf directory.txz directory. The -J argument to tar means that as it reads the directory and creates the tar archive, it passes the bytestream through xz in compression mode. Then you have to tar -xJf directory.txz to get it back, which passes the .txz file through xz decompress, and then untars the resulting original files.

You can also use xz directly to compress an existing file. Say you created an uncompressed tar archive and then decided to compress it:

tar -cf directory.tar directory
xz directory.tar

(Note that this erases the directory.tar file, replacing it with directory.tar.xz.)

If you want to uncompress in two steps:

unxz directory.tar.xz
(directory.tar.xz is replaced with directory.tar)
tar -xf directory.tar

Filesystem compression is easier for saving on disk storage, but you have no access to the files in compressed form. You can only ever see the uncompressed bytes. If you xz a file, you can send it in compressed form to someone else.

 

[onefang]

But picking another timezone doesn’t actually hide where you live. You are most likely to have a pattern of times for commits that will point to your real timezone.

I keep seeing people claim this, but the problem is that you are assuming hobbyist programmers ONLY program 9 to 5.

I spend a lot of time programming and stuff at all times of the day and night. I often have to be up at crazy early mornings or late nights to chat with people on the other side of the planet.

Even state actors could be working nights, or faking it.

Then there's night shift, I have been a night shift programmer in the past.

Edit: Ninja'd by Xign.

 

[VividVerism]

I keep seeing people claim this, but the problem is that you are assuming hobbyist programmers ONLY program 9 to 5.

I spend a lot of time programming and stuff at all times of the day and night. I often have to be up at crazy early mornings or late nights to chat with people on the other side of the planet.

Even state actors could be working nights, or faking it.

Then there's night shift, I have been a night shift programmer in the past.

Edit: Ninja'd by Xign.

Yes, but someone working a hobby project will most likely work sporadic hours with most of them on weekends and evenings. A consistent pattern of weekdays for roughly 8 hours per day at roughly the same time every day points to a regular workweek.

It could be a false flag, but it's very unlikely to be a hobby project.

 

[foobacca]

Interesting interview with Andres Freund on the Oxide and Friends podcast: https://media.transistor.fm/e2538f7d/1567b6fc.mp3

Some interesting details about what helped him notice it in the first place, the process of raising the issue and even some things the attackers could have done that would meant he wouldn't have noticed it

The podcast is by the company doing the oxide cloud computer thing, https://oxide.computer/