How Much Memory Does ZFS Need and Does It Have To Be ECC?

external systems and how

much

memory

do you

need

for ZFS and wait, I know it must be that you

need

for each terabyte of storage, you need at least that many gigabytes of

memory

, there is a formula, it's a little more confusing than that and that answer has been in the top of Google search results I think it's too long because that's not the most accurate way to do this. What we're going to cover in this video is that you don't need that

much

memory for ZFS, but you do use a lot. of memory, you will get better performance from ZFS.

I

have

a couple of videos where I dive into the And, which of course is caching, the more memory you add in there, the better the arc cash, therefore the better the performance. The video is linked below where I hopefully demystify the ways the cache system works. Kind of a side note is what about ECC? Isn't that a requirement for ZFS? Isn't my data at risk and I'm putting all this data away? I know that in the hands of a bit flip and destroy everything, that's not true either. I

have

a video talking about ZFS being a cow that copies the correct file system and don't worry, data integrity is fine even if you're not. using ECC, is it better to use ECC?

That's a general yes because, from a computer science perspective, having error correction memory is a good thing, but it's not a necessary thing to do this, so I wanted to clarify them right away. Now let's dive in. We're going to talk specifically about secure Nas, but this applies to GF ZFS in general when you configure this for the amount of memory you need for your ZFS system. Okay, we'll start at the low end. This is my low point. end of the Intel Atom system, this is what I call my purple Nas because I had a bunch of purple Western Digital drives that I put in here, four of them to be specific, let's look at the very fast storage here and we'll see that I'm only using four terabytes of the 22 terabytes that are still free so a little less than 30 terabytes of storage on this it just keeps backups of things so performance is not an issue it's backups of my videos so I don't mind worry how It will take a long time to recover them, but if my main Nas crashes, I have a second copy, but I restarted it a moment ago because I just want to delete it from the copies I made when I backed up and saved files.

It is to show you that with eight gigabytes of memory it is still not even used. Now I have 5.5 gigs free. If I start copying files back and forth, it will use some memory and if we go and look at the services I'm using on it to get the data back and forth, like rsync. I have an rsync job that communicates with it. I have SMB if I need to copy some files. SSH, which is part of facilitating ZFS shipping, so my main video editing safely copies all data. I come back to this one pretty regularly because, hey, why not?

It

does

n't require much; only differential copies are made so this job runs frequently but it is not performance intensive and as you will notice I have quite a few terabytes of storage which

does

not match the amount of memory and the system is fully functional and working well, aside from being slow, but it's slow because that Intel atom, more than anything else, is combined with some old, spinning disk drives. talk about my video editing system when you get scale vs core there are some differences specifically because scales built on linux by default don't use all the memory they will only use about half of it so we have 64 gigs of ram in my edit Of video. system because I could probably use all the videos more and in fact that's why they are under load, my videos are recorded directly into this system, they are cached, this makes it very fast when I have to embed those videos into the edition. software, so it comes straight out of cash and because I modified the parameter, I'll leave some notes in the description below to tell you to use 50 gigs of cash, it's using 50 gigs of ZFS cache, there are always files moving back and forth back in my Video editing gives me much better performance now, even though these are SSD volumes, they won't peak at their 10gig Nick, but they will be much closer to hitting the 10gig Nick when you're pulling things out of memory.

So this is where having more memory helps a lot. Now let's move on to one of our production mainframe systems in my office and this system has 120 eight gigs of memory and is pretty much used all the time for ZFS caching. There are some services. This happens because we run multiple S3 targets for different backups that arrive here and this system benefits from having a lot of memory. You would benefit more from more memory. Well, we still have 5.2 gigs for you, so maybe not much more, but this is using everything. the money to serve the virtualization targets that talk to it or some of the backups and restores when I do backup restores, they get from this when I have my virtual machines running NFS, this is the storage target for them and because NFS and VMS are going to be a lot of frequent rights or frequent reads, more specifically for database things that are being pulled, you just cache everything and serve it very quickly.

Now you have 27 spinning Rust drives, but the cash will supplement and really make those frequently called files, which is a typical use case, served extremely fast combined with a fast processor to transfer the data back and forth and this system actually works and has similar IOP performance to an SSD array; -a comparison will vary depending on the workload, but this is an example of how having a lot of memory helps a lot when we have clients that have large virtualization systems and use TransCore as a large target for those where we load as much memory as is reasonable. to your budget budget is usually a constraint at some point on how much to buy, but loading a system with 512 for a truly net system is not uncommon because it will give you really good read performance for all those files accessed with frequency or in the case of like a system that runs 80 or 100 virtual machines on several different hypervisors and pulls the data here or in the case of video editing, we have a lot of customers in that space that are more than just an editor like I, if you have a team. of editors that pull files from memory will always be faster than pulling them from drives, as I said in a video.

I use true and answer as an example, but it applies to any system that uses ZFS. "You have to have a lot of memory, but memory will greatly help the performance of any system that is running ZFS, so it allows you to use more memory, which means better caching of the video. I delve into the details that ZFS is really smart." We'll take all the memory and all those files from one place to another and send them there so that any service that requests them can access them faster, but if you don't have them, it will just take the drives out and work at any speed. operates depending on your system configuration, let me know what you think in the comments below or to engage further in this header or in my forum for deeper discussion, thanks