Mutual TLS | The Backend Engineering Show

In this episode of the background

engineering

show

I would like to talk about a relatively popular technique called

mutual

tls or mtls. It's not a new thing per se because it's been there since day one, I think, but it's been popularized specifically by certain use cases around microservices and this is what I want to talk about so we can talk about that, obviously, we need to talk about what tls is correct, which is the current most ubiquitous way to perform authentication in tls. I'm going to talk. about what kind of disadvantages is that associated with, can I talk about what is mtls, what is

mutual

tls really, is to talk about the advantages, the problems that it solves and obviously nothing is for you in software

engineering

, introduce something which comes with its own sets of problems and that's what we're going to talk about we're going to talk about the disadvantages or things to keep in mind when you implement mutual tls, let's get into it welcome to the

backend

engineering program with your host hussein nelson and this is our long and deep conversation on a topic.

I like this podcast because it's the only time I can take a topic and go as deep into that topic as possible. You can't do it in 60 seconds or in a few minutes, you know you need time to explore these options that you know and the conversation details, and I totally understand that some of you are not fans of this long form content, you prefer to know shorter videos eight to ten minutes, but some of you are like This type of long form content and this is, I guess, for those types of people, so let's get into it and start by discussing what transport layer security actually is.

It was invented to encrypt the communication between the client and the two-party server. They exchange symmetric keys using a key exchange algorithm and once the same key exists on both machines, they use that key to encrypt the communication. Any packet they send is first encrypted with that symmetric key, and as a result, no one in between can see it. the actual plain text is correct and better yet no one can modify it because if you modify something that is the most dangerous thing you know on the internet you know it can lead to bad things because if you can go to a site and I change it to it makes you go to another site which is very dangerous and this used to happen all the time with unencrypted plain text http.

You see if you go to http google.com in the old days you know a git request will be sent and that get request will be translated so you know the forge layer is converted to a tcp segment before we establish the tcp connection once we have that in layer five, know the session file descriptors and all the uh session variables and session metrics, then we move on to creating a segment, you know. and we send that segment, that data is sent to a segment, it depends on the mtu and the size of the segments and their size, depending on you know the protocol that we are going to use and basically depending on the slow start algorithm, how large That initial number of segments that we can send is also controlled by something called the slow start and congestion avoidance algorithm and I talk about that in my fundamental networking course.

Is here. Check it. Go to Dot Hussein network and also Dot Com for a discount. The coupon from that link is automatically directed to Udemy with that discount. It is applied so that the get request is put into a segment and that segment let's assume for simplicity it's just a segment that will have problems, it will be in an IP packet and it will go through your router first, my router is out there, that is the first destination, you know it's also going to start doing networking and sending it over the Internet, and the first thing it's going to go through is going to be through my ISP, so whatever it does.

The shipping goes through my ISP, so my ISP is in the way, it has to go to my Internet gateway. My ISP's autonomous system connects me to the wild internet and there are other routers that control it, so everything I sent goes through my ISP, which means my ISP sees everything I sent fine, except if it's encrypted , in this case it is not right. I'm sending the plain text and I get the request by going to Google, so if my ISP or anyone in between happened what happened. be on my route, okay, it sees my data, that means they can change it so they can pause that ipacket instead of forwarding it like normal routers would.

They may respond to me by saying, "Hey, actually, no, this is where you're supposed to." to go to http code 301 redirects to this website and this website is owned by the attacker, so the fact that you went to google.com redirects you to a site that is not google and now you are completely under the control of the attacker and that The act of the redirect is to establish another TCP connection and go through that and that new site could be encrypted, but the fact that we didn't encrypt at the beginning is the problem, so what we do is always try to encrypt the connection. first and for that we use tls which is awesome so sending a plain text http is just a bad idea in general because this could happen so we invented https so I always start with https let's always use tls as always We encrypt first, so in this case, you establish a communication and everything is encrypted, but that's not what tls is only used for. tls is actually used for authentication.

What does that mean? It means that if you are logged in to google.com, how do you know you are really logged in? visit google.com tls can guarantee that you will see when you establish an https connection on http, correctly http https, you make the tcp connection with the server, you get the server, then you turn around and then you do a tls, you send the tls hello that has this key exchange that we talked about and it says okay, use this algorithm and use this key size and use this and it has a bunch of parameters and inside there is something called sni, there is an extension in the tl called name indication from the sni server that tells you Hey, I want to go to google.com, but you could say I'm saying: didn't we just make a DNS and ask for the ips from Google?

Is not sufficient? No, actually, for many reasons, because once you get that iPad. you lost the knowledge that you are connected to Google, you are connected to a reverse proxy and then you will send that sni and from that sni of that text that says Hello, I want to connect to google.com in the extension in tls the server is responsible for returning the which we call a certificate, so that certificate contains the name google.com because that's what you requested, say "ok" and it contains something called a public key and it may contain a signature and that signature is signed by the private key corresponding to that public key, because that's the only way we can guarantee that we can return that certificate and if you got the certificate, you can verify that this public key actually participates in this certificate, how well it takes the public key and decrypts that signature and if verified correctly that means whoever signed this could have the private key and now we know the certificate is valid but that's not enough because literally anyone can do it, they can create a public key and a private key and claim to be facebook. com and create a certificate as a self-signed right that is not enough to verify that it is accurate because it will be, how do we prevent people from pretending to be Facebook correct servers that prevent them from being Google?

What we do is we trust a third party to someone higher up who can be trusted and that's basically the certification authority, so what Google, Facebook and any website says, okay, here's my public key. You don't get my private key at all, here's my public key. Trusted key certification authority worldwide. Please sign this for me so that whoever sees the certificate can trust me. This is my actual public key that belongs to Google and certification authorities usually do their due diligence to verify that it is actually Google. I'm going to put a file on your machine, let me try to access that file and once that happens, okay, I trust you.

I'm going to use my private IP, my private IP address, my private key, my right to sign your public key and this. It's a test and I'm going to include my public key as the certificate authority and my name and all the information so you can see how big this can get. Now the question is fine. I send these two pieces of information to the client and the client will take the certificate. I trust the certificate, but I don't trust that the public keys belong to Google, who signed this. Oh, here's part of it. It says the name is certification authority.

Let me check my certificate store as a client. that looks good, the certificate authority is right, there's my store, let's encrypt it, it's really reliable, that's enough for me, I need, I don't need to go further in the chain, that's enough or that's not enough, why I couldn't find a shark. encrypt the certificate authority, sir, did he really trust you? Who knows, because maybe this certificate authority simply does not exist in my certificate. I am an iot device and I don't have a large search store. I only have basic things, for sure. Well, the chain that the user sent actually included who signed that certificate authority and it could be another certificate authority and that could be another authority certificate until we get to the certificate, it basically has nothing on top of it, the root certificate and once We have to get to what must exist in the store if it does not exist, the client rejects the certificate and thus basically passes the verification so that you can see that the chain including the entire mask can be expensive and this is another topic that I am going to talk about in another podcast where these certificates are actually not compressed by default, you get all this stuff from the server and if you understand how the network works you should put them into segments and if they are too large use too small. these have to be sent as multiple segments and as a result can delay the tls handshake obviously for two or three and check who cares but if you're huge you know your scale like a content delivery network you know which adds up well.

So that's the authentication part, so the advantages of this, you see, you'll notice that we only authenticate the server, you know the server sends us the certificate and we never send anything about the client authentication, that's how it works for the most part. of the Internet today, we request to connect. to a server and the server sends us the certificate so we can verify it and trust it, that's how most communications work and it's very useful, you know, because you see if I performed as an attacker a DNS poisoning attack, what did I intercept the DNS? it's mostly unencrypted until we get to doh or dot as the attacker, assuming I'm in the root like at Starbucks and I did a r poison and pretended to be the gateway and all the iv packets now go to me instead of the real router, so I'm in the root and I can see and change everything, so now if I see a DNS going to Google, I will immediately respond to the user.

I won't even bother sending that DNS request, so google.com is actually one two. three four and one two three four ips is actually executed by the attacker's server, so now the next thing the user will do is establish a TCP connection was one two three four instead of the Google IP address, pretty cool, so now TCP connection is established because it's just that TCP has nothing to do with authentication. Hey, this is the address of my amplifier. I'm going to go online and just connect it to the attacking server. Okay, no big deal, we didn't exchange anything yet.

Continue with the following, what do I do immediately after tcp? I do tls correctly, I send the tls hello and say hey, I want to connect to google.com again, so this is another one. The second time we mentioned the hostname, we actually mentioned it three times. times if it's http and the first in the dns the second is the third is the hostname and the http header so I'll include that in this and the server name indication as plain text there are efforts trying to encrypt that esni, i think it failed, approach the client, hello encrypted clan, close trying to encrypt everything and mostly succeeded.

I don't think I have much information on that, but you can go deeper into that, we might as well talk about it. in the future, so now I submitted that and said, hey, I want to go to google.com. I know where I'm going. I have it stored. I wanna go. I still need to go to google.com. Okay, that attacker. You will receive that the attacker will receive a request to establish a secure channel with itself, but it is the clients that request the google.com certificate. Do you think the attacker has that right? The attacker does not have that right.

The attacker cannot. You can easily create a self-signed certificate claiming to be google.com, but the client will immediately reject it because it's self-signed, you won't find it in your certificate and there is no certificate authority that will sign a certificate claiming to be google, that's not google, nor facebook nor anyone except diginotar, which happened a few years ago, maybe 10 years ago, where the certificate authority will think what compromised and started signing certificates for google and gmail and it wasaccess this by using network settings but again I imagine there are some limitations to reach why we reached this level and I am not aware of these limitations yes because it is definitely cheaper performance wise and I am not going to say uh, infrastructure maintenance effort to really prevent just the connection, just block it, write it, say, hey, this network, this subnet can't access the API gateway subnet and you can block it, you can put yours own, you know, virtual. the network, so to speak, it's true, Amazon allows you to do that, I think that's right, so you can do all these tricks, but I think there is always an exception and these exceptions are the most unpleasant, so maybe the mutual management of zlas is easier in that case, but I could be wrong there that's why that gives us actually the problems of mutual dls.

What are the disadvantages of mutual tls? Sure we have always trusted servers, but how expensive it is to trust clients. How much configuration do I need to do? How much management? How much maintenance do I have to do on this client? make this mutual, I can't imagine anything that needs to be done more and that would definitely cost extra unless there is a way to somehow make the certificate management certificate store centralized for both servers and clients and I think that this could be how expensive it is to run this store by updating that reissue certificate correctly, all of that has a cost sure, I guess you can argue that this is the cost of doing business, but it is a cost regardless and I would, I would let it In the hands of the development engineer who actually deals with this on a day-to-day basis, perhaps you can tell your war or horror stories when it comes to managing certificates.

I don't know if it's difficult or not. I never did this before. The problem is that we have many more clients than servers, so yes, with servers, we have the certificate store and we generally trust the certificate authorities more and anything that is signed by this certificate is enough and if we trust the certificate authority should be enough. Well, if we have a certificate authority and as long as we have a certificate, that should be enough. Once we know that this certificate is signed by the certification authority and we manage to obtain a certificate from the certification authority, which makes me think now. about another attack chain where an attacker managed to spin up a container on the network and requested a certificate from the certificate authority, I imagine because now you need to test, I mean, how do you prevent that in itself from looking like it has to be So? done by some other, you know, administrative process, almost, you know, no one can just apply for a certificate, otherwise anyone will apply for it again, get it, so this is really interesting, you know, thinking about all these things and in another problem we can.

Think about it, even if you have tls mutual rights, you have mutuality across all your microservices, and you have x services well. You have access to the API gateway and only those guys can talk to this and let's say your API gateway was pushed to github or any stolen session key that was leaked, you know, you're an attacker and now you want to use that key of API. You might say you can't because you're an attacker. You can't actually just connect to that API gateway internally. Because you don't have access to it, then you don't have mutual tls, but what if I talk to the load balancer's reverse proxy?

It says Hello little dancer, here is my API gateway. I'm off the grid. completely outside and I sent a request with a header with an authentication, enter with my API and get with our post. I don't think we have mutual tls by default with reverse proxies publicly, it's like anyone can connect to that public load balancer, so if the fact that if we blindly take that request and forward it to an internal microservice, that microservice turns around and establishes a tls session with the actual API gateway within the microservices network and at that point it is trusted, so now it's almost as if the attacker has been proxyed through a mutually trusted microservice tls is completely useless because how and this server will authenticate and even though it will, it will retrieve the results from the database and we will be authenticated and then we will go back to the respondent that the load balancer will respond to.

The attackers say, "Hey, here's the data you requested, cool, so mutual theorists are useless in this case, right, because it's not publicly exposed all the way you know, it's completely useless in this case, so that you need to find a way to have a way to do it." tag the key with a certain IP address or a domain so the load balancer or reverse proxy can do a bit or the public API gateway, if you are right I still say the API gateway is in reality which exposes the public law while the authentication is carried out. could happen in a later client so yeah these are some of the issues I can think of from mutual tls obviously nothing is perfect but I always try when there's new technology like this new in the sense that it's popularized.

I tried to do it. I guess I thought: What is this going to buy later? I'm summarizing this. We are going to finish this program very soon. What can really go wrong? Because you always know we are proud as software engineers when we invent something. all of us never almost never talk about the possible disadvantages, right, you even see it in rfcs, right, sometimes they talk about this and this, okay, we know these current limitations, sometimes we honestly just don't know, this is something Honestly, it's like we do. I just don't know, we just discovered these attacks as we go, I mean even if performance and optimization improvements are made, this happens gradually, why do you think we had hdb109 and then one or and then one one, then we have this performance? that we turned off header compressions on one because we couldn't figure out how to solve the crime, you know, crime, crime, capital, I forgot where it means and then we finished db2 and we ran into another problem between ending in b3 and fast and now.

Fast is perfect, no one talks about perfect, but fast is the best, and I'm sure that in a few years, if not already, people will find limitations and fast limitations must be three problems. I think those already exist because the fast is built. in udp, udp statelessness is very scary, you know, replay attacks, you know, fragmentation, ip fragmentations, nasty things like that can happen, uh, protection at the top level obviously exists, but it's scary when things are stateless because anyone can just send it, there is no state. persistent on the server like tcp where hey, here are your streams, this is where your window size, I know you can send me something okay, I need to verify that it exists in my session, you know, yeah, and butcher distributors are no different , I think it's good.

I think overall it's very good, but I think we should really pay attention to the limitations and just because something is good doesn't mean we have to always implement it just like we can't implement it in mutual stories on the internet. Luckily, it doesn't make any sense to implement on the entire Internet, but I might like certain services. I can see us in the future with my phone somewhere in my pocket. I think I can see something like a client certificate per phone or paired device like my Mac, my phone would have a client certificate and I would register this client certificate public key before with Netflix and it says Hello Netflix, only these certificates can access to you.

Okay, these are my public keys, right, or maybe I can. I create a domain and I sign this domain with Netflix, but then Netflix has to manage all this searching, which is something they never did, obviously, they didn't have to do it, but maybe it's safer, right? it's no I didn't read anything about passwordless authentication is this this maybe you're right because that's that it looks like this this could be the right passwordless future if this is what passwordless is and you know something completely different because like Hey , I can register a certificate on the server and mutual tls is the future maybe but good luck like the server supports mutual tls and certificate stores and yes maybe it will happen but I don't think it's impossible for certain sites Great websites, right?

It's definitely more convenient it's like, oh, I don't care about passwords, credentials, tokens anymore, bye, the moment you open your phone, if you have something physical, that physical thing with you, that's your key, that's your key public, I trust you and you have your private key. obviously secure key and obviously all of this should go to this, what is this chip called? I keep forgetting about this hardware I had because it was bothering me. I had to Google it. tpm trusted platform module, which is a module that is completely isolated from the operating system, it has its own CPU cache and that module is responsible for storing and issuing private keys and all that trusted platform module, so yeah, I imagine the search will be private parts, at least a private process, that's funny, the private sections of the certificates will be included in the tpm, but yeah, this is all fascinating and when I do this, you know you can tell I have a very general scheme and, as I speak, I am surprised by what I say because I am discovering new things.

It's so funny because I don't have time to sit and talk to kids and life anymore, but it's like when I make these videos it's like I'm having a conversation with someone and while I'm having a conversation, I'm discovering new things and you'll be surprised because I recorded this video like before and I threw it in the trash because while I was recording I was thinking about all the different things and the videos like can we come? It's like what's happening. you guys are all over the place I'm all over the place in this video too but a little bit better I hope you enjoyed this episode I'll see you in the next one again guys yeah go to network.json.com for a discount coupon on the basics of my network to pack effectively and of course then check out my other courses also head to head nasa.com there are all the courses that support the program correctly considering becoming a member becoming a member support this program there are more 60, I think it's a giveaway, so 70 exclusive members-only content, you get the slides for free, you get a special badge, fancy stuff next to your name, you have special icons that you can use in the comments and in the chat, we'll see you in the next.

Guys, stay awesome. Thank you so much.