Virtual Reality Engineer Explains One Concept in 5 Levels of Difficulty | WIRED

Hi, I'm John Carmack, I'm the CTO of Oculus. I work in

virtual

reality

. Today I have been challenged to talk about a

concept

in five

levels

of increasing complexity, so we will talk about

reality

and

virtual

reality. What does technology allow us to do today, what will it allow us to do in the future and if that should be our goal to get closer to reality, then do you know what virtual reality is? Yes, it's simple, it's like a video game, except you feel it. like you're in the video game, that's actually a very good description.

The idea is that if you have a system here that can make you see what we want you to see, then we can make you believe that you are somewhere else like on top of a mountain or in a dungeon or under the ocean or in Minecraft, yeah, or in Minecraft, when you look at a TV on the wall, it shows up as a picture of a mountain or something, how can you tell that it's not just? a window and there's something else behind it because it doesn't always look quite right if you have a static image of a person on a screen and you move like that, it doesn't really change and it's interesting, those are things that we have to solve Discover the ways to Avoid it to trick yourself in virtual reality.

We need to find out when you look at something in reality. How can you tell if it's real or not? Have you been to a 3D movie where you put on the little glasses, um, yeah, so what? They do the trick for that: if you ever go to the cinema, take the lenses and look at them, you'll see that it's blurry, where there's actually two images that are showing at the same time and what those little lenses do. Do they let one eye see one image and the other eye see a different image so your eyes can say Oh, it looks like I'm looking through the screen or something is floating in front of it in the VR headset?

What do we do? Are there actually two screens or a screen split in half so that it draws a different, completely different image for each eye and we make sure that each eye can only see the image that you intended and that's what can make things feel like they are? I have this real depth to them that it's something you can reach out and touch and it doesn't feel like a flat TV screen. Try to look. I look over there and focus on his face. Can you see me waving my hand without turning my eyes no okay so at some point you can probably see it right now so without moving my eyes this is a little difficult tell me how many fingers I'm holding up you just say it and I say it you see even in "as if." you extend your hand and you focus on your hand, then your foot would be blurry because there is that difference there and you can switch between that, then you can focus on your foot and your hand becomes blurry because your eyes can see different amounts of details. in different places, so we hope that in the future the hardware can be like that, where we can make a screen that places a lot of details right where you are looking and every time you look somewhere else, it moves the details there, so we don't need to render a hundred times more than what we have now.

Figuring out where you're looking is a pretty difficult problem. What we try to do is take a camera and look into people's eyes and then try it. to find out if the eye is looking here or up, we're working hard on things like this right now, so hopefully we can have a virtual reality that is as detailed and realistic as the reality we have around us, but it's going to happen a lot. time before we get to know where we can really fool people. Do you have a basic idea of ​​what latency is? Yes, my understanding of what latency is is basically the time delay between rendering and rendering. different points, so it's basically a delay that occurs in all parts of the system.

Monitors can be large, as you know, consumer TVs can often have 50 milliseconds or more of latency just on the TV part and then you have the processing on the computer and this all adds up to the total latency. In my opinion, latency is the most important part of VR because if you have that compensation, your body is no longer submerged and you get that motion sickness that can attract a lot of people. outside of the experience the games feel really good, they have that feeling that it happens instantly when you know you're doing something, it really is a testament to this type of technology and how it's developing and how over time you'll just be able to include more pixel density on those screens and it will be much more immersive, like 30 years ago you had desktop computers where you spent whatever you had on that, but there was always this idea. you can spend a million dollars and buy a super computer and it will be much faster and that's not really true today for scalar processing when you just do one thing after another, a high end cooled and overclocked gaming PC is the most fast that it exists. the world is within a very small delta, yes, for some things you will get some power, you know, the IBM power system may be a little faster, but not by much, so if I look at this and say we should be five times faster you know what you're doing you can't just say do everything faster as a developer you're making a trade off I can put my effort into making this more efficient or making it more fun and fun usually wins for very good reasons so there's something of good judgment and trickery in designing things because you can always design a game that just won't work well.

I mean, in the old days games had to be designed just as precisely today. You have a lot more freedom, you really can, it's just the hardware because older games run in 8 bits and that's all the data you can have. Yeah, any crazy idea you come up with now you could probably make a pretty good video game, which is. It's a wonderful, wonderful thing, yes, a lot of freedom, but virtual reality makes you give up a little of that freedom. You don't have to do so many crazy things to end up being responsive and high quality.

I know it's very interesting, you know, an interesting topic is what are the limits of what we can do with virtual reality. I'm pretty happy with what we have today, what we can show people and say VR is cool. People get an incredible response. but we are still clearly very far from reality, that is reminiscent of realism in our history and how realism was a response to romanticism and realism was meant to capture the mundane everyday lives of people you know and not idealize any of them. their activities in any way and I think that's really important for virtual reality.

I think it's like a rite of passage for any kind of our technology, mainly in virtual reality. We talk about visualization in optics, the visual side of things. but we should at least dial in the other senses and haptics is something interesting about virtual reality, it doesn't really have that aspect of touching things, you can move your hands, you can do everything, but it's a disconnected experience because you know you don't I don't know the real strength there and I'm pessimistic about progress in haptic technology in almost all other areas. I'm optimistic, I'm excited about what's to come, but I don't have any bright vision for where we are.

We're going to revolutionize haptics and make it look like we're touching things in the virtual world, so you know, I tried the demos where there's that brla, there's one that has waves like audio waves, I think they appear and then you can put. run your hands through that and feel the waves every time you're supposed to feel bubbles or any kind of force field or something and that's pretty interesting. I've seen some pretty interesting things you can do with audio that you can reduce a lot. from the storage, I guess in the power you would need to power a huge scene, you can just mimic the sounds of those scenes that are actually there and then not build them, for example a USC professor would have the sound. of a train passing by without even playing the sound and you would feel like you were deeply immersed in this world without having to build such an expensive scene around you, so I think they are quite important, and that is a possible improvement of the quality that is still on the horizon is that when we do spatialization we use the head relative transfer function hrtf to make it sound like it's in different places, but we usually just use this kind of generic here like the average human hrtf function and it's possible Which, of course, if you're average then it's perfect for you, but there are always people who go to extremes where it doesn't work very well and there may be better ways to let people try their own perfect product. hrtf, which can greatly improve audio experiences, it comes down to all these trade-offs that you know with display and with resolution, it's one of those things where if people have a bad experience, it trumps everything else, It's really difficult to build.

You go back to trusting people who haven't done VR before, but it's easy to break that trust every time they do it, but there was a big concern about it at Oculus and the term floating around internally was poisoning the well. They were very, very worried. I mean, for a long time there was a fight about whether Gear VR should even be made because the concern was that if we allowed a product to come out like Gear VR that didn't have those things, if someone saw it and it was bad, it made them sick, it made their necks hurt. eyes, then they would say I would never try it again.

I tried it that time and you know it was terrible and there were legitimate arguments about whether it was even a good idea to do that. and it turned out that yes, it's obviously better to have all that stuff, but you can still do something that's valuable to the user without it. It's strange to be at the beginning of a medium like this. I'm very excited to see how the filmmakers do it. approach content creation in those things, especially if they already have experience with the traditional medium, mainly today. I've been talking a lot about what we can do, what's possible, what we think might be possible in the next few years, but really on a professional level it's rather the question of the wisdom of what we should be doing, that's one of the things. that we're trying to figure out, from an artist and a storytelling perspective, what are the things that are going to make this significantly different from what we're used to. like a TV on our wall and we've been finding a lot of things, aspects of virtual reality that I think do a lot of that, things that allow you to feel presence first and foremost, where you get lost and you have to remind yourself of this. things that aren't actually happening and things that ultimately allow you to embody other characters, things where you can really change your own perception of yourself and play with neuroplasticity and learn things that are strange and unique as an

engineer

.

Of course, I love the quantifiable things I like. saying: here's my 18 millisecond move towards the photon, here's my angular resolution and I'm getting better. I'm doing the color space right, but you can look, not too far back, where you say we have Blu-ray DVDs with this incredible resolution, but More people want to watch YouTube videos at really bad Internet video speeds, where there are things that , if you deliver value to people, then these objective amounts may not be the most important thing and, although I'm certainly pushing as hard as I can. a lot of these things that potentially improve the experience in every way or maybe just for videos or the different things, I don't think it's necessary.

I've mentioned that I think generally my favorite titles on mobile are completely synthetic. They don't even try that, they just go and do like flat shading with lightmap and I think it's a lovely aesthetic. I think you don't end up fighting all the aliases while getting other titles that we are going to be high. technology with our, you know our specular bump maps with harshness and you have aliases everywhere and you can't keep up with the frame rate and everything is problematic, while some of these that are clearly you know very synthetic worlds where they are nothing more than these cartoon plans. shadowy things with lighting but they look and feel good and you can believe that you are in that place and you want to know what is around that monolith over there.

We did a project called Life of Us, which was exactly that mentality we had. adopts the low-poly aesthetic and just the vertex shading and we end up realizing that you can embody these various creatures and transform and when you do it with the co-presence of another creature, another human, um, it becomes a totally magical journey and you Don't even think for a second that you actually throw out the whole idea of ​​photorealism and accept that reality as it is. I think it actually helps you calm down a little bit. The ultimate goal of reality, of course, in computer graphics people have pursued.

Photo realistic, you've known this for a long time and we've basically done it. Photographic realism. If you are willing to launchEnough, you know, discrete path traced rays on things, you can generate photorealistic views. We understand light very well. Of course, it still takes half an hour per frame, as always, or more, to render the different things, so it's an understandable problem and given infinite computing power we could do it in VR, however I have made a point. people in the last few years is that we're running out of Moore's law, I mean, maybe we'll see some wonderful breakthrough and you know, quantum structures or whatever or bandwidth, but yeah, but if we end up going down the path We're going to double and quadruple, but we're not going to be 50 times more powerful than we are now, we're going to run into atomic limits in our manufacturing, so since I'm trying to tell people to start buying again in optimization, to start thinking about buying again. a little bit more creative because you can't just wait, you're not going to get to that point where you're really looking at those higher degrees just waiting for computing to advance, if we want this to be something used by a billion people, then We need it to be lighter, cheaper, more comfortable and there are constantly new ux innovations, like in Google Earth, the way they removed your periphery as you zoom and move, and also give the user the action to decide where is. going and looking we constantly see people coming up with ways to break away from the paradigms of real reality and then introducing mechanisms that work very well, you know there are tons of opportunities for the synthetic case where you want to be able to have your synthetic fantasy world where they're all reasonably simulated computer-created creatures, yes, but of course we have them, you know, we still don't simulate people well, it's a difficult problem, we've been banging our heads. against for a long time, I think we're making progress and I wouldn't bet on that being resolved in 10 years, but maybe 20 years, because it's going to take a lot of AI, it's going to take a lot of machine learning. where it won't be a matter of us analyzing every micro expression that people make, but rather we will take every YouTube video ever made and run it through a huge apprentice who will figure out how to make people look realistic, yes, absolutely there.

It's these crucial thresholds that you overcome, you know, technological obstacles and suddenly that opens up a whole world of creative potential, but I think, to your point, we need to solve real human and social challenges and turn them into opportunities to figure out how. This technology fits into our lives. I still believe the magic is out there, we haven't found it yet, so someone is going to figure out the formula for you. I feel like I've felt little corners of me, yeah, oh. You can imagine the utility behind this to create a world or you can imagine the power of a story that would be told to you in this context and I think a lot of it is just choosing those that bring them together in a meaningful way and then creating something.

That's really bigger and more intentional, but now I've been joining real people in VR and I think we also have different

levels

of connection, like the audio side, which represents leaps and bounds in terms of the nuances of personality and The humanity. when I hear people laughing and joking and you know I'm really enjoying something, we'll get into that soon, we're in a land of macro gestures where I can wave and say, you know, thumbs up, but when we get into micro gestures and we really have a sensation. from facial reactions and other things, I think then we will have some incredibly rewarding moments spending time together, so virtual reality right now is pretty amazing when you look at it, it's things you haven't seen, but we're just getting started.

The next five years, both technologically and creatively, will truly take this medium to a place you never imagined.