This Server CANNOT Lose Data…

When you make as many videos as we do, you need a lot of fast, reliable storage. And our main editing

server

, Whonnock, has checked all those boxes for years. He's a great little

server

. It is built with high-quality components and even has an attractive appearance. But as our team grew, we got to the point where even one minute, a single minute of downtime, costs more than $50. And that's just on payroll. Now, in practice, the way to mitigate

this

is by adding redundancy. Now our units are redundant. We have 20 drives with

data

splitting, but the problem is that they are all on one server.

And I'm sure you can see where

this

is going. It's been over a year in development, but it's finally here, the final form of Whonnock, and I'm calling it... Whonnock 10, because it's the last Whonnock ever. High availability Wow! Wow, 10! I told you this like 10 times. Nobody even knows what high availability means. It means it's Linus-proof. Go ahead and unplug one. Do it, go ahead. Well, I should probably tell you what's at stake before you do that. Each of these two large twin boxes has four full servers inside provided by Supermicro. We sponsor all of this. And they are configured with Weka, a redundant NVMe-first file system.

In this configuration, it should withstand two entire servers going down without anyone noticing. Except we moved the entire team last night without telling anyone. And we're in the middle of the workday with a ton of high-priority videos in progress. Do you really want to try it right now? I have not tried it. Alright, here we go. Well. What can go wrong? I want to say a lot. Naturally, a big part of a project like this is the software, the stuff that will be responsible for distributing all of our hundreds of terabytes of video projects, Word documents, and Linux ISOs to the multiple machines we just showed you.

But we can't install any software until we have some hardware. So why don't we start there? Meet the Supermicro Grand Twin A+ AS-2115GT-HNTR server. Despite its ordinary appearance and unexciting name, it is anything but ordinary and it is very exciting because inside this 2U there are four independent computers. But for what we are doing, four nodes? Please, we want eight. Inside each of them is a completely independent motherboard, 384 gigs of memory, an AMD EPYC Genoa processor with 64 cores, two M.2 slots for redundant boot drives, six PCIe Gen 5, two and a half inch NVMe slots on the front and We have I/O on the back.

Now, this part here might be a little confusing at first glance, but that's because not only do we have USB, but we also have two full PCIe Gen5 by 16 connections here, along with display output and power for the entire server. This all slides into the chassis, which contains a really cool modular backplate assembly that we'll get to in a minute, and then goes, thanks Jake, to the back of the server, where you have a management port, a single USB port for each server. No. There are two and they are shared. That? I was just about to ask, because we also have a single VGA.

Do you see the button? For two servers there is no way. Does this button toggle? Yes. And well, before we talk a little more about that, look at these power supplies. Each of these is 2200 watts, 80 plus titanium, which seems like a lot, but when you're potentially driving four 400-watt Epic Genoa CPUs along with plenty of RAM, up to 24 NVMe drives, and eight network cards, well, it seems downright reasonable. , No? Are there 24 units? It just can't be. Yes. Six by four, yes. Six times four is 24. And of course, that's just one of them. We have two of those.

And that means that in the event that one of these fails, the system should be able to continue running uninterrupted, which is a big part of the high availability goal we have for this deployment. Speaking of high availability, let's move on to our network cards. Each of those 16 PCIe Gen5 slots that I showed you earlier ends up in one of these OCP 3.0 small form factor intermediate slots. And what we are putting on them are these 200 gigabit ConnectX 6 cards from Mellanox. Sorry, from NVIDIA, okay, these are the older Gen 4 ones. Therefore, they will be limited by the slot speed of around 250 gigabits per second.

But if we had newer cards, that means each of these nodes could do 200... plus another 200, 400, up to 800 gigabits, which of course would be a complete waste for us, A, because our workload doesn't You can take advantage of it, and B, because our switch is only 100 gigabits. I'm sorry. Of course, the two ports are still useful. We have redundant switches. Except there's kind of a problem here. That's still a single point of failure. In a perfect world, we would have two single-port NICs. So if a NIC died it would still be fine. But since we have so many nodes, we don't really care about an individual node.

You know, they could have a boot drive and die, or a NIC and die. We still have an extra backup. How many nines do you want? I mean, I don't know, one would be nice. Nine percent? Jokes aside, it's a good point. If we were designing this correctly, there would be many more considerations we would have to make. As if the power going into the rack has to come from two independent backup sources. Connectivity with our clients would also have to be redundant. Connectivity between all systems would have to be designed in such a way that no matter what goes wrong, everything will stay up and running.

And being realistic for us, we're not going to go that deep into this because our goal is better than what we had before, which was a single machine with its own built-in redundancies, but other than that, nothing. Now we should at least be able to

lose

one complete machine of these eight. We can reset one of our main switches, totally fine. Two machines out of these eight and we can still limp along. I mean, limping is a bit of a stretch. It's going to be very fast. Now, typically, if you buy a Supermicro machine, they will pre-build it for you.

They will validate it for you. You can even ask them to pre-build one or more entire racks with these elements and then validate your application on them before they ship them to you. In fact, we have a whole video we made about that that was sponsored by Supermicro a while back. Of course, this is LTT, my friends. So we will assemble it ourselves. Do you like that twist of the screwdriver on the server? Don't worry, I won't fail. I will never fail. Look, I could do this a hundred times and it would never fail. No, it's okay, it's okay.

Okay, we have seven more. Anyway, for our CPU, we chose an Epic Genoa 9534. This is a 64 core, 128 thread. CPU monster. It will make a maximum boost of 3.7 gigahertz. It has a quarter gigabyte of level three cache, a 300-watt TDP, supports DDR5 memory, up to 12 lanes, and supports a whopping 128 lanes of PCIe Gen 5. Originally, we intended to use 32-core chips, but they were exhausted, then. Free update. We're so lucky. Compared to previous generation AMD EPYC CPUs, Genoa is a big step forward in terms of IO performance, making it perfect for this application. And in the long run, I mean, if we have all the extra CPU cores and a lot of RAM anyway, why run Weka bare metal when we could install Proxmox and then use the other cores?

I don't know, high availability, Plex server. Yes, or Linux ISO. More realistically, it would be something like Active Directory. Yeah, which we really don't want to do right now because if you run Active Directory on a server and it goes down, you're going to have a very, very bad time. But if you run it on multiple servers. Yes, it's great. Normally server CPU coolers would come with their own pre-applied thermal paste, but since we are doing it ourselves and... If you look c

lose

ly, this is not the first time this has been installed. We'll use... Well, thanks for that.

A chunk of Honeywell PTM 7950. This is freaking awesome. It has excellent heat transfer properties and can withstand different temperatures. Seriously, I don't remember how many cycles there are. It doesn't even just vary, but it likes many temperatures. Huge cycles for a long, long time. Now available at lttstore.com. Is that big enough? Does that cover all CCD, CCX and CCW? Oh, there's a second piece of plastic. I am stupid? Is there a second piece of plastic? No, there is not. Should I put one in the refrigerator? No, no, no, it's totally fine. I have done this many times.

Oh, she's perfect. Look at that, see? Easy. I recommend storing it in the refrigerator before using it. Alright. To ensure we get the most out of our CPU, especially in this high-performance storage workload, we'll fill our 12 memory channels with 32-gig DDR5 ECC DIMMs running at 4800 megatransfers per second. That's a total of 384. About three terabytes of memory. That? In the eight. Oh. Each of the cables Jake is removing now is a PCIe 8 cable that powers two of the drive bays on the front. But the reason he removes them is so we can install our bootable drives. These are consumer grade.

Each system receives two Sabrent 512 gig Gen 3 rocket boosters. And it's not because they are particularly special in any meaningful way. They're not even that fast by modern standards. But in our experience, they are reliable enough and fast enough for what we are going to do, which is simply boot our operating system from them. Cinematic magic. All other nodes are already built. So what do you mean by movie magic? Supermicro built them. Oh, I thought you built them. Supermicro built them for you. I took it apart. Well okay. I took that one apart. There are no more secrets.

Yes. No intrigue, no mystery. You know that what is still mysterious is inside here. In fact, I had never opened this before. (SPEAKER_02)Oh, okay, let's take a look. Court! Holy shit. Oh, those are the power supplies. Yes, this is great. So the entire computer is cooled by four fans. No way. There are the two fans on the power supply, and then these fans on your, what is this called? As an IO module, I think that's what they call it. Look at the blades on this thing, they rotate counterclockwise. You're serious? Is that what you're looking at?

No this? The most delicate of spaghetti. Oh Lord. There aren't even any connectors. No. Each of these cables is soldered directly to the back of the OCP 3.0. That? Yes. For storage, we are installing two of Kioxia's fast CD6 Gen4 NVMe drives on each node. We have one of seven terabytes and another of 15 terabytes. They're kind of placeholders for now. And in the long term, we'll move to something close to four 15-terabyte drives per node. But the drives we want to use are currently occupied by, oh, that project. For a top secret project related to pastry. So that will have to wait.

The good news is that when those drives are available, Weka supports live upgrade and downgrade so we can simply rip these drives, swap in the new ones, rip, swap, rip, swap, rip, swap. As long as we don't do it all at once. Are we ready to turn these things on? Okay, there's a lot going on here. What's that? Is that a switch? Yes. Hey, look, now you can see the button. Oh! That's great. Boop, boop, boop, boop, boop, boop. What you're hearing so far is just the Nvidia SN3700 32-port 200-gig switch. OMG it even says Mellanox on the front.

I know, maybe it's an old review sample and demo unit or something. No, we got it with the $1 million PC and I'm pretty sure it was already Nvidia at that time. Can you hear that? Do you hear it getting louder? Yes. Wow! He's just excited to see you. This is the Weka panel. Maybe if I go here, cluster servers, we can see all of our servers. We have two albums by and then choir. This is a very interesting part of how Weka works. It's not like TrueNAS, say, where it just uses all the CPU for whatever you're trying to do.

They dedicate and like to delimit specific nuclei for specific tasks. For example, each drive has a core. So we have two unit containers. That means two cores. One complete core per unit. Yes. Damn. Yes. It also has compute cores, which like spoof computing and inter-cluster communication. And then there is the interface, which you don't necessarily always have. The front-end kernels manage the connection to a file system. So if you only had drives and computer, you wouldn't be able to access the files on this machine. Then you would have your backend servers, right? These would run units and compute, which is the cluster.

And then in your similar GPU box, you would run just the front end. And that would allow the GPU box to connect to the backend cluster servers. Oh, sure. But the backend cluster servershave. Here is another of your servers. Is working? It's cool though. For reference, you're not supposed to do this. You have to shut down the system first, but we're just trying to simulate it crashing. Yes, a terrible catastrophic failure. I can't believe how well he handled it. See all the lights. They never stopped blinking. Many thanks to Supermicro for these amazing servers. Thanks to Weka for creating this crazy software.

Thanks to axel.ai for the amazing AI detection. If you liked this video, maybe check out the video series where we build our almost three petabytes of file storage, which we call the mother vault. That thing is amazing. And we show it to you and now it's faster. Oh, and thanks to you for being an amazing viewer.