What Everyone Missed About The Linux Hack

We have just experienced the largest exploit in the history of open source software. I've never seen anything that encompasses everything from the social engineering side to tricks for really wealthy skateboarders and just taking advantage of the entire open source ecosystem to be able to create a very well hidden exploit, so well hidden that only a random engineer from Microsoft because they were comparing their SSH connections to previous similar security issues, honestly, they weren't that similar if we look at things like J logging or even those things. As a bed of heart in the open SSL world, those bugs were honest mistakes made by maintainers who were just trying to write good code and things slipped through code review that weren't built into exploits, they were just honest mistakes that that people committed,

what

happened here was "It's not that a bona fide maintainer was exploited, he was harassed, he was manipulated, and he was tricked into building trust with an individual who then built a very rich skateboarding trick and then distributed it all over the world." world if you're running Linux or Mac OS there's a good chance you have the stuff exploited here on your systems and if you're running a new enough version of Debian you might even have the exploit to make this an absolute mess. and I want to do my best to cover this responsibly.

So instead of just being a soy JS developer and pretending I know

what

security is, I'll call someone who is much smarter than me to talk about the side. of security. Now you're probably wondering how an open source project is a project where

everyone

can see and read the source code. It is compromised by a malicious backdoor. That is a very good question. The way it was done was quite ingenious and it was done through these two binary object files that were pushed to the repository, not as source code, but. As just data blobs, these two binaries are injected into the build process and when on desop disk are converted to bash scripts.

Now what this bad script ends up doing is taking these layers that hide the obis data inside a nice big compress. lzma and extract the internal evil object file and then make it part of the build process so that the linker now depends on the evil object file at build time, as far as the affected systems are concerned this only matters if a couple of cases are true. the repository you have is not from gits The source control is not from gits Version control, it is a release tarball that was from GitHub, they did not include this code in Version Control to keep it hidden and you must also use x86-64 and a Linux gnu variant for this to compile into Li lzma as a backdoor.

It is also important to know that the backdoor is only enabled if the following is true if the term is not set as an environment variable and if the binary being run is the user Espin sshd very important even though there is a backdoor in a widely used compression library it only matters for sshd now what does this backdoor actually do? Here I have the object file placed in gidra, the disassembler, by the NSA. and there is a function here called get cpu id get cpu id is a function that is normally just one inline line, it is a line that the compiler inserts inside get cpu id which is called by the linker and executes all the malicious backdoor functions.

Now the reverse engineering community is still working out the details of this backdoor, trying to figure out what are all these named functions that would normally do compression stuff but actually do backdoor stuff, what they do and how they work, but if Do you want to follow me, I'm working on a video right now for this topic. It should come out as soon as we know more about the backdoor, but that's all for now. Thanks for hanging out if you hadn't guessed. That little low level piece knows a lot more about the security side than I do and it's interesting as you can tell there's a lot left in this video and I don't really plan on talking more about the security side.

This amazing diagram was posted by Frogger and I highly recommend it if he is interested in the security side of things. What I want to emphasize here is that in this diagram we have this big part at the beginning that goes from 2021 to where xlo was introduced in 2024, which is a very small part of this diagram, most of this is focused on how the exploit works , how it was introduced, how the backdoor and the bash file was introduced and all of that goes together, there are much better people to cover that than For me, what I want to cover is the part that I don't think is getting enough attention, which It's the craziest trick I've ever seen in my life.

The social engineering part. This individual did not simply put evil into a project that someone else was doing. In execution, they exploited the existing maintainer to take over a project with many users and be able to do horrible things, so how did they do it? I read this phenomenal article by Rob Mening that delves into the manipulative side of the experience here. from open source maintainers and this is really horrible, let's talk about how the nature of open source can be exploited in this way. Rob Mening posted a pretty interesting article that was originally like a Twitter thread talking about how this is an Open Source flaw.

Source itself to some extent and I think this is a really interesting take that really shows the risk here originally on Twitter about the XC Liel CMA vulnerability. When I finished writing it, I realized that I had a real-world slice of open source interaction that I deserved. More attention, there will be many analyzes of the XE lib lzma vulnerability. However, I found that most skip the first step of the attack again. That's why I'm making this video. This is a really important piece that the original maintainer burned and only the attacker offers to help, this is the key, there was one maintainer and then there were two maintainers, one of which was exploiting and was waiting for the opportunity to take control and Finally the first one burned down and now only the bad actor remains. unprecedented planning and execution like we've never seen before in open source.

Surprisingly, someone found a file with an email thread that captured the state of the world just as Step Zero was taking place, let's read his words first, we start with a reasonable request. Reasonably, the question forces the maintainer to address his flaws. I use failures in quotes here because A, the maintainer doesn't actually owe anything here, so he hasn't actually failed and B, I know exactly how he feels, it feels terrible to let his community down like this. is directly from that email, XC for Java is still maintained. I asked the question here a week ago and have not received a response.

Oh, I hate these messages so much. I hate these messages so much. The amount of times I've gotten stuff like this about randomly building or working on this it's the worst feeling being too busy for a week to respond to things and the response isn't oh I hope you're okay maybe we can chat soon Oh, aren't you working on that? This is already the most passive-aggressive thing and it is the worst feeling to receive messages like this. This is absolutely a key point of the beginning of this maintainer's burnout. The maintainer recognizes that he is behind and is struggling to keep up.

This is a cry of pain. this is a cry for help, help will not appear in this thread again, it is very very real and painfully common, yes by some definition at least if someone reports a bug it will be fixed developing new features definitely not He is very active, he has a frown again. very understandable oh here we are introduced to our attacker Li May in the same message not the help you expected certainly Gan has not helped me and could have a more important role in the future it is clear that my resources are too limited So, something must be change in the long term.

This is when the attacker offered to help. Instead, a helpful UNH consumer says useless things. This is exactly where these types of email threads go. Progress won't happen until there is a new maintainer. The current maintainer lost interest. or no longer cares about maintaining, it's sad to see a repository like this, given that this exploit appears to be an intentional attack by Gan, in case Jagar Kumar is considered complicit in actively encouraging the original maintainer to abandon it, no I'm sure I'll see this useless client again soon, interesting. I like the implication here. Is it possible that Jagar Kumar isn't a real person and was doing some crazy social engineering to try to burn out this maintainer faster so that he would be more likely to be willing to give?

First of all, there are about 200 IQs in the project, not only on the exploit side, but also on the social management side. This is a people exploit, first and foremost, as great as security. Inevitably, the maintainer tries to fight back. the stress of burnout differently I tend to get angry which ends up coming across as sarcastic, however this reaction is heartbreaking, yes I feel this. I'm definitely the sarcastic type, but I've seen other really good maintainers just burn out and it's the worst. painful, I haven't lost interest, but my attention span has been quite limited, mainly due to long-term mental health issues, but also due to a few other things.

It seems that this particular poor developer was targeted because the package was simple and should have been easy to maintain and he probably assumed that when he built it, but due to struggles that existed outside of the work he was doing he didn't feel he could maintain it and some another party pushed and pushed until they gave up and the maintainer also reminds

everyone

how the world's software is built now. It's also good to note that this is an unpaid hobby project and as always, the xkcd dependency comic is more relevant than ever. All modern digital infrastructure.

All the madness we are building is half. held up by some random project from some guy in NE Nebraska, thanklessly holding it since 2003, yeah this is the first time we've seen at this level, someone looks at this chart, looks at this comic and says, "You know what I'm betting on, I can get it ". person to abandon it and let me hold this in place and that's exactly what this attacker did and again this was all in 2 weeks, someone filed an issue and got no response for a week, made a really rude comment, a week later they came back and made Another rude comment, unfortunately there are definitely real people who do this, on one hand I think this is the attacker doing it, but on the other hand I've seen people be this rude on Open Source before and I wouldn't be surprised if this was real. person you ignore the many patches that are rotting on the mailing list right now you choke your repository why wait until 540 to change maintainers why delay what your repository needs okay now I'm convinced it's the attacker just the tone of this is the attacker this other person, what is this for?

I can't express how angry this makes me feel for the maintainer yes, honestly, part of why I'm pretending this is the attacker is probably because I can't imagine a human doing this and I don't want to believe they would. I'm probably in some degree of denial right now, where obviously a real person who isn't an attacker could do this, but to be right with humanity I have to pretend. this person is intentionally acting maliciously to make all of this happen. Another really good point Nick just made from moderation in a security perspective. It is now dangerous not to ban rude people like this from shouting it from the mountaintops in the future.

Cite this example as the reason. You're banning people because now you don't have to just say I don't want to deal with you. Now you can say that dealing with you could cost the security of our package. Bye, be nice if you want to talk to me and you. I hope we can take that little silver lining out of all of this, so let's get back to this reasonable applicant, he decides to come back and make demands. I'm sorry for your mental health issues, but it's important to be aware of your own limits. I understand. It's a hobby project for all contributors but the community wants more but the community then forks it and then forks it if you're not happy with the speed it moves move it yourself it's open source you can fork it whenever.

Same thing Rob killed it with this article. Definitely give it a follow if you haven't. This is even better than I expected. Read the last sentence again. The community's desires must be fed by more consumers. The maintainer's needs, of which there are clearly some. the important ones are ignored, yes, they are no longer reasonable. The applicant also offers a suggestion notice that there is no offer forhelp, there never is, they always complain because they want you to do the work for them, why not move maintenance from XC to C? so you can either pay more attention to XC for Java or hand off XC for Java to someone else to focus on that for C.

Instead of trying to maintain both or trying to maintain both means neither is maintained well. I don't love the suggestion here. I think there is a way to say this that is kind and thoughtful, but not on a mailing list, it is in private conversations after you have built some trust with the person, but this is so far the one that seems best intentioned, but still being a Really dumb thing to say on a chain mail list that everyone can see like this, then the maintainer explained the reality, finding a co-maintainer or handing off the project entirely to someone else has been on my mind for a long time, but it is not something trivial.

For example, does someone need to have skills, time, and enough long-term interest specifically for this? It's also a great point. Everyone seems to think that at least people who aren't true open source contributors seem to think that you can just pick a random person and help them maintain your project as one of the things I get asked the most by open source maintainers. how the hell did I find so many cool people to build things like create T3 apps with me. I shouldn't even say it, I should say it, because they're doing all the work. The reason.

I can do that because I have an amazing community of amazing people like you who have hopefully already subscribed to this channel. By the way, press the button below. Subscriptions are free, you should consider that, but I have this amazing community of people who get high. This community doesn't have many newbies because being a newbie makes my videos hard to watch because I'm not going to teach you the basics of things. I'm not even going to tell you what the definition of a word is. I'm only going to talk about what interests me, so I have a community of people who, on average, are much more technical than the typical community member, and I also keep a close eye on who they are. make things more interesting and attract them to build a small more united community within the chaos that we are making, that is only possible because I have this Mass M platform with hundreds of thousands of subscribers and millions of views a month and even then I can only find between 5 and 10 of these people, if you're a random block, you don't have that platform, you don't have those people to trust for that kind of thing, which is why the people who are in Surprisingly, I often go to this position to ask someone like me: Hi Theo, I need help maintaining this project.

Can you help me find people to do it? It's actually very hard to do if you're a random developer just using the package, it might seem oh. just find someone else it's not that easy yeah so if you push someone to just find someone their bar won't be that high it takes skills and knowledge to write software and while a lot of skills and some knowledge will transfer when working on a new A software project inevitably requires the development of new skills and more knowledge. Some developers aren't fungible cogs that you can swap in and out of things all the time, yeah, like you can't just swap most developers in and out of things, especially if they don't get paid.

They have to care, they have to understand that they have to be productive and they have to know how to manage a community to do open source maintenance, most developers are not one of those things in the four, so it makes sense that even They like the 1%. of the developers are probably not cut out for open source maintenance the third party in the email with complaining consumers do not offer help while continuing to make demands, only the attacker remains Gian may have a bigger role in the project in the future, he has been helping a lot off the list and is practically a co-maintainer already has a smiling face, yes, and I want to be very, very clear, not only do I not blame the maintainer here as much as the original maintainer, whatever the opposite of blame here is , that's how I feel, I'm really sorry and horrified that his mental health was exploited to do something as terrible as what happened here and if anyone ever talks about this maintainer for what happened here, I need you to hear what What a bad person you are if you blame them for this because they took advantage of them to do free hard work for everyone to use.

They went above and beyond and they were just doing everything they could to make sure that this thing that people depended on stayed well. They didn't do anything wrong here. They did absolutely nothing wrong. here I love the summary here this is really good I totally agree this is a microcosm of things that if you are a maintainer you have experienced and how bad it is that's why I'm getting so hot because I've been a maintainer in the past and I still help maintaining a bunch of stuff, one of the most thankless jobs I've ever experienced, we just don't get it, it's actually fun to go from open source to YouTube because I'll do one little thing and I'll get a ton of praise on YouTube.

I'm going to do something big and get nothing open sourced and I really want to thank the original maintainer here. I know he was just cursing someone who was talking, but he went above and beyond and as a result, his GitHub account was suspended. G is the attacker, but he was following lass, so I could see from his followers that lass was suspended and I go to the opposite, also Gia, so the attacker was suspended. That makes sense. Lose suspension. There is no GitHub, if anyone. watch and listen if you don't have a good reason to suspend this account release it now it's horrible that someone who had their mental health exploited and did nothing wrong and has no harmful compromises on their account is getting any for this. not at all free L, this is crazy, he actually updated his blog and wrote some details here and it's mostly just a list of facts, but I want to at least quote this because he deserves to be featured here because he's doing everything he can and more. credit to him OG Prodigy just found a message from the maintainer that I think is really valuable to read here.

Hello, I read the post on the open wall. I was on vacation and checked email. I spent time with friends and them. I'm also at home right now, but I thought I had to spend some time on this since I check emails. I'm really tired, but I guess I should do something right now. A longer investigation on my part can probably only begin on Monday or Tuesday, this sounded too serious to ignore. I feel so bad for this maintainer, holy shit, he was literally on vacation hanging out with his friends and he's trying his best to move on, being super honest about it all the time.

This is breaking. My heart is straight, this sucks, God, that's so sad. Again, if anyone gives this guy anything, you'll be on my list forever. This is the brief description of him because he just wanted, as an official source, to provide information here and I have the respect of the Master. Him for doing this to find time, even when he is doing many other things and trying to enjoy his Easter with his friends, but he is still here talking about it. Big credit to him for that, also the pull repository for the actual project has been removed from GitHub, which again, if the repository was removed, his account should be reset.

The fact that his account is not restored is terrifying to me. I really hope GitHub makes the right decision and brings it back soon. This page is short for now, but it will be so. I will update as I learn more about the incident. Most likely it will be again during the first week of April. It's not just about getting more information, he's trying to take a vacation, but he seems focused on this like someone who's been in the middle of one. Crazy drama in the past, it's hard to even just sit with your friends and eat food without that being on your mind constantly.

It's the worst feeling in the world. Something much bigger than you. Everyone wants your opinion. You know it's like something. a lot about you but you can't really do anything it's the worst feeling in the world and I have a lot of sympathy for lass for what's happened here the git repositories are at this url here because again they've been deleting other places so is making sure they are accessible XC, t.org, DNS name has been removed, XC projects currently do not have a home page, this will be fixed in a few days. Facts, this is a cve for it.

XCS 560561 released archives containing a backdoor these archives were created and signed by Gan the archives created by gan were signed by him all the files signed by me were created by me it is good to say this that his signing credentials were not exploited because there has been some skepticism about things like Twitter that an account was exploited and people were making fake commits and someone even said this is a reason to sign your commits, you should sign your commits, but this is completely independent of that. The accounts of both Me Laru and Gian have been suspended.

This sucks this sucks so much x.org the dnsc name was hosted on gith pages and so that's why it's ok if you're seeing this I'll be more than happy to help you in every way senses with hosting this, if github is not going to reset you, I will personally make sure this can be hosted, I will do it out of pocket if necessary, you deserve all the support from the community, you can get more and I'm happy to being able to give you My own money and time are at stake for that because I am truly mortified as I continue reading this.

I have only had access to the main tu.org website. I got the tu.org repositories as well as related files that only Gan had. access to things hosted on GitHub that included this site because it was like a subdomain that went through GitHub and this is the only thing he had access to on the domain. This is a very good option to do as well so that we know what we can and cannot trust. I'm amazed at how helpful this little post is and I'm so grateful you did this even if you're not taking the break I wish you'd take again.

You are a legend. You're doing this better than anyone would expect. A lot. Someone who was bullied out of the project. Well, in my opinion, this is a textbook on how to handle when something like this happens. You have now written the book on how to do this. true and I hope other maintainers who are seeing this and are mortified can at least learn lessons from how well you've handled this and how to prevent this from happening in the future. As it turns out, this is another IRC message that came from the original maintainer and there's a really good quote here.

I wanted to highlight how crazy it is how much Gia helped. I still need to get more data to exclude that it wasn't her account that was compromised, etc., although the evidence I've read is abundant. Gia was already inclined, in fact he helped, so he was playing all sides of this. This is one of the two craziest IQ Warfare manipulating social engineering and software

hack

s I've ever seen. I can't imagine anything else that comes close to this and I feel really bad for the poor maintainer who was exploited this way for laru on behalf of the open source community and software as a whole.

I hope you know how sorry we are, this sucks and you did nothing wrong here and if anyone here or anywhere talks about you, do it. don't let that be your problem let it be ours the community needs to do better here and we as a group need to stand up for what happened and do everything we can to build a culture where this can't happen in the future because it wasn't like that. an individual issue, this was not an engineering issue and it certainly wasn't a code review issue, it was a community issue and we failed this maintainer and we have to do better, that's all I have to say about this .

I'm horrified, so, yeah. I'm going to email the maintainer and let him know that we respect him a lot and don't harass him, don't spam him with stuff, but if you see him out there, let him know that he did well here and until next time. peace time nerds