Why a Forefather of AI Fears the Future

Hello everyone, thank you for joining us for one of our ongoing conversations about developments in artificial intelligence and related subfields, where we've been talking to some of the world's biggest thinkers, some on the research side, some on the more business side, some that focus on Ethical issues Political issues Sociological issues Of course we're going to touch on all those kinds of questions in our conversation today, but we're talking to one of the greatest computer science researchers in the world, who is really responsible for having pushed our understanding forward in this rapid process. developing Artificial Intelligence Arena and I'm talking, of course, about Yosua Benjo, who is a professor of computer science and operations research at the University of Montreal, he is also the founder and scientific director of the Quebec Institute of Artificial Intelligence and he is also the co-director . from the Canadian Institute for Advanced Research's Learning, Machines and Brains program in 2018, he received the Turing Award which those of you who know will recognize as an Essence being the Nobel Prize in Computer Science, so he won the traveling prize.

I shared it with Jeffrey Hinton and Yan Laon and they won it for his fundamental contributions to the field of artificial intelligence, so it is my great pleasure to include Yosua in our conversation here. Hello, how is he doing? Thanks for inviting me. Thank you for joining us. Look, I just wanted to jump in a little bit before your award-winning work on tour and everything that's happening at AI, it's just good for our audience to get a sense of you, you know your own development, you know your background, so I'm guessing you were where were you born. in France it's that right, yeah and uh and that's where you were one of those science nerd kids or what was your passion in the early days um my family moved to Montreal uh when I was 12 and um I was already interested in science, but above all During adolescence, my interest in computers especially, but also in physics and mathematics, grew and I started in the 80s and started programming on the computers that we had these days, and that explains many of the decisions I made.

Later at University and as you progressed, did you study computer science throughout or were you on the fence between different passions within the Sciences? Well, I've always been interested in understanding humans and, in particular, human intelligence, so I was interested in a variety of areas, but because I got into programming and, you know, did reasonably well in mathematics, I focused more on computer science, so my degree was in computer engineering, where there was more, more physics. that if I had done a computer science and TR program, there was no undergraduate program in computer science at that time at my university and it was only graduate studies, so I did a master's and a PhD in computer science computing, but the big choice happened when I had to choose what to do for my grud studies and that's where I was lucky enough to have read some of the first articles by Jeff Hinton, who, you know, has been a role model for me and really became passionate about this question of is it there? a connection between what we do with computers and human intelligence and there's something like the laws of physics that would explain intelligence, that would be cool, yeah, uh, so, you know, looking for that has been a really great motivator and, I mean After decades of thinking about intelligence, whether embodied in a computer or embodied in a flesh-and-blood human being, the perspective emerges that human intelligence is something special and singular or is simply the natural culmination of a computer system that becomes sufficiently sophisticated. last thing, I think we humans tend to overestimate our uniqueness in the universe and of course we are everything, but clearly intelligence is something that can be seen in very different forms in nature and of course more and more. more in computers in different ways, true, not that there is one way to be intelligent, yes, but as we understand the principles better, it becomes very clear that it is something much more general than ours and when you say that there are different ways, Now in a certain sense I mean, obviously, we can look across the animal kingdom and see many different types of intelligences specialized for certain tasks that allow a certain living system to prevail in its particular environment, of course, but we can all imagine that it is about of a spectrum of living intelligence that we are familiar with when we are finally at least among the general public.

I know you guys in the field have been developing these things for decades, but now we're finally finding a different, seemingly different kind of intelligence with the big language models and image identification in production, even in video production. , with the sound of a prompt, is kind of mind-blowing to someone from the outside, but then when I hear people describe it and I've read a little to get an idea of ​​what's happening on the inside. seems radically different from what we do, so would you consider this a moment where we finally encounter an extraterrestrial intelligence compared to the intelligences that have been on this planet or do you consider that even part of a Continuum is clearly an extraterrestrial intelligence, but?

It is also very close to ours in many ways, it is close to ours for two reasons: one is that it really is the inspiration of human intelligence and, in particular, neuroscience, yes, that has driven many of the decisions that we have made. in the last decades. The second reason it's pretty close to human intelligence is because we're training it in human culture, yeah, sure, but I still mean we clearly see things that are very different and clearly there are gaps that aren't just. different, it's also weak in some areas where you know there are things that a 10 year old would do easily that CH GBT has problems with, yes, but it's useful, I mean, in a sense, if you're just trying to understand. things like understanding intelligence, it's useful to look at those differences to see where AI is better than us, where it's weaker and maybe try to understand the reasons for these things and if you said where they're weaker, it's like in the Domains. planning and reasoning is where you would put the EMP part, yes, so there is a broader category that I have talked about for about six years that you could call conscious processing, so everything we do consciously. course reasoning planning, but also things like counterfactuals or being able to evaluate how certain you are about a thought or a judgment that's coming from you, yes, I mean your ego, your ego, could be in the way, but if so, you know you are capable of doing it. to evaluate that, then that kind of epistemic humility is something that humans can do in their current machines, they're not very good at that, um, yeah, being able to consider many different interpretations at the same time is something that humans can do and, again, ego can interfere. with that, um, but the machines are not that good and uh, that's actually one of the dangers of these systems, uh, they could be wrong for sure, but maybe the most important thing is what you said, so planning the reasoning combining knowledge in a coherent way with clarity.

We see that, you know, movies have problems with that, they are improving, yes, their main weakness, but I guess I always like to have an idea of ​​​​the people on the inside, you know, on the outside, obviously, November 2022 was quite shocking for many people. and, frankly, maybe it's still shocking, it's not the reaction to describe what's happening today, but you know, the other night I was playing with one of these systems, you know, my wife and I were tasked with creating a certain type of document and it was just amazing how with one message we could get something that you already know pretty well, so on the outside it feels pretty amazing on the inside to have pushed the boundaries and been responsible for the kind of developments that allow these systems to work.

Did you see this coming? or it was a little surprising to you it was also surprising it was surprising so let me clarify the methods as far as we know because unfortunately there are a lot of things that we don't know, but based on the information we have, the methods that different companies use are not that different from the things that have been documented in academia. What is different is not so much the algorithms as the scale at which they are trained and the size of the models. This is well studied now and was anticipated long before GPT chat came along, you know, because we were seeing that as we got the new Nets to train more with more data, they were constantly improving in all the metrics, but I think few people interpreted what it meant at some point. in terms of hey, this thing can basically manipulate languages ​​well or better than most humans, they still know things that they don't understand well, they don't reason well, but language, this wasn't something we expected to figure out that quickly, yeah, frankly , etc.

Does that leave you with the feeling that, going back to your initial comment about how humans tend to understand what we can do inside our heads, it's the ability of a system like the GPT to do the kinds of things that we used to think that You know only we can do it. Does that mean it's hugely special or that we're more common? I mean, how do you climb the ladder? Yeah, by the way, these systems are even smaller than your brain in terms of uh. The number of operations is difficult, it is difficult to calculate, it is difficult to like because because your neurons operate with very low precision, it is difficult to know exactly, you know what the correct mapping is, but they are probably still significantly smaller . although you know that maybe the next generation next year or so will be closer in computing power than a human brain, so we're not far away and it's also plausible.

I mean, Jeff Hinton has this argument that they're actually more efficient, in other words. that at every synapse, these AI systems are more efficient than your brain, so it's not just because the synapses are more precise, but also because they are digitally encoded and that gives them advantages when training them, that the brain has to deal with this kind of noise and if you want irreproducibility, uh, the substrate that these machines don't have to deal with, then you know it's an interesting development because I mean, in my Arena physics, the big moments were singular creative leaps. usually by one or a handful of individuals, Einstein with special general relativity, you look at where he was in his understanding before he wrote those papers and where he took the world after them, and there are a handful of ideas that are impressive in the power of he.

Quantum mechanics is a little different. You had a generation of scientists working together and it was a peaceful way to progress, but it was still these ideas that seemed like huge creative leaps, not just a change on some kind of scale. or data, but you are talking about a radical change that simply comes from the size of the data set that is being used and that seems very different to me, yes, so that allows you to move to scale, but of course it depends on conceptual ADVs that have happened in the previous three decades, for sure, or even the four decades, actually, because we were talking about ideas, for example, that Jeff Hinton talked about in the mid-80s and early 80s, um and, um, For example, a notion that I worked a lot on and that I built as one of the first neurologicalists. net language model in 2000 and is based on the idea of ​​representing symbols with vectors which is now of course everywhere in these systems, yes, and there are mathematical reasons why this is a good idea that were not obvious before that you knew it.

In the old symbolic approaches to AI people weren't even considering things like this, the logic was just based on symbols and the symbols weren't based on high dimensional vectors like we're doing now, so this is an example of an idea. that has been developed. You know it in several steps, but yes, that is really transformative and there are other things, for example, something that comes directly from the inspiration of the human brain because when you have a thought, let's say you think about a cat, your brain has a particular pattern. of activations, so that's basically what this represents with Neal Nets.

Another thing that comes from your brain in terms of inspiration is attention, so one of the big advances in neur net architectures came with a tension mechanism, so in 2014 my group. introduced that kind of uh control the tension in your networks that in 2017 gave rise to Transformers right and we know where it went so they basically piled on all these attentions on several levels, many levels and it's really changing the game in terms of competition . So I'm just giving these examples as simple and fair ideas that can even be theorized, not in the same way.way like quantum physics or relativity, but you know, you can really make a case for why this might be a good thing, right? and that's been necessary to get to where we are today as far as we know, so as it goes on, we'll talk more about this a little bit later just to get an initial idea of ​​it.

Now you know. obviously two main prongs that one can imagine that of course are interrelated one prong is that the data sets will get bigger, the computing power will get faster the other prong of course is the new techniques and ideas creative and innovative ideas that you and your colleagues around the world are developing. Where? Which one will attack us first? Is it the I? I mean, what is the situation right now? You mean what bottleneck will occur. Yes, say it in the negative direction, but yes. The amount of data written is something we are approaching the limit of.

Maybe we are. Don't know. the numbers because they are also hidden, but I imagine that we are in a small percentage, maybe you know 10% or between one and 10% of what is reasonably available, but it is not that simple because the data from the highest quality. and what is left is of lower quality, so it could be that we are quite close to the limit, since the speed at which these systems have been increasing the size of their data set is much slower than the speed at which the Humans are producing more things, uh culture. It grows but not so fast, right?

So what about synthetic data? Can we have synthetic data? The system does not, but it does not have the same, it is not new content, but there is an area where there is still a lot of space, which is video. true, and it's very rich and the reason there probably hasn't been as much progress is simply because it's much more computationally expensive. In other words, the computing power to process a high-resolution video like a movie is a lot more to say. the same two hours of reading a book, yes, much more, yes, of course, and then how has all this work worked out over decades and the most recent rapid developments?

How has your view on human creativity changed? I mean, when I think about you, you know? a great language model in the back of my mind. I imagine there's nothing really creative there, it's just combining things that exist and finding new ways to put them together using statistical analysis of which word is most likely to appear next and so on, it doesn't feel really creative and However, I look at other things, from Shakespeare to Einstein, and I feel like, wow, that's a profound creative leap, something parallel to the question from before, how do you change opinions?

I think I need to change my views on this, so let me explain the connection of what I was talking about with planning reasoning and so on. First of all, creativity is a complicated thing and there are many aspects of creativity, so there is an aspect of what you call mixing things, which is already creative, which is being able to bring together a lot of aspects, like when you are asked to draw a image, yes, the images are new, it is a new combination of concepts and you get the corresponding image, okay? So it's kind of an easy way of creativity, but that's the way most of us do most of the time, right, we're not Einstein.

You know, creating relativity every day, unfortunately, yes, but that part, you know, that kind of creativity. matter of course it changes the world let me give you an example of where machines have had that kind of creativity when we are very good at planning and reasoning so look alphago yes what is happening here is different from the movies. very different, it is a very different type of neural network algorithm in alfao, there is an explicit search, it is stochastic, etc., but it is a search, it is a search in the space of sequences of movements, which is a reasoning that is about of sequences of things that are centered together correctly, so when you search, you allow yourself to explore new combinations in a way that is aimed at achieving something like in science, we are trying to find something that explains the data better or in a new way and There's a different kind of creativity, it's not just about putting things together that you already knew, it's about finding a new solution to an old problem, let's say, and of course, I found ways to play completely new MH strategies that humans didn't expect. , he invented new ways to play that are better than what we knew well and he is able to do that because he is doing that exploration that looks in the space of combinations of correct Concepts, which is much more than good.

They give me a set of aspects that I will create a new text or a new image. Here's this optimization that looks for something very specific that has a particular property in the case of science, often what we're looking for is a very compact description that explains a There's a lot of data, so a very brief physics, you know, the equation It surely explains a lot of things and that is extremely valuable, something that we can quantify mathematically in the context of searching for theories that explain the world correctly, but with Alpha Go I. What I mean is that the thought that comes to mind is that the system works within a fixed set of rules, the rules of the game, and simply looks for what those rules can generate, whereas the creativity we value most in physics is the contribution that changed the rules that gave us a complete idea, Can you envision a system that has that kind of flexibility completely?

Before I answer that question, let me connect the two types of AI that we've been talking about, so you have the movies that are trying to figure out how the world works, but not the way I like, but anyway that's What they're doing, they're developing some understanding of the relationships between the entities that they see in the data and you have the more traditional reinforcement learning. like alphao, where the rules are given, so how the world works is given and it's simple, yeah, and what a lot of researchers are doing these days is good, how do we combine these two things?

Can we have an AI that can figure out how things work? in the world and use that to find new solutions to the problem of how to achieve goals and we still don't know how to do it well, but this is that we have like two pieces and people are trying to put them together. now there is a particular type of search that is like now, coming to your question, there is a particular type of search that is not just achieving a goal like in general, but achieving an explanatory goal, which is what scientists do well, so I mentioned finding a theory that is very compact and does a good job of explaining the data while searching the space of theories like thinking in the space of mathematical equations, one that has this very good score in terms of being simple and explains a lot, which It's called later beian, by the way, that's a search c also like what scientists search for, but its goal is not oh, I want to know how to find the way to get home, its goal is to answer these kinds of questions to find in space. of you know strings of symbols that we use in mathematics any really good answer to a difficult question I'm sure that could come up this is something that interests me a lot there is a whole subfield of machine learning called AI for science where people among other things are trying to explore these types of questions and there have been successes so far that give you confidence that this is a field that in itself could in the not too distant

future

take off and somehow do my work and at least maybe the jobs. from some of my immediate colleagues around here something secondary to what we currently think, yes, there has been quite a bit of progress, but we are far from human scientists, so what is the timeline to basically achieve AGI, what people call general intelligence artificial AGI, at the human level?

Cognitive abilities, of course, no one knows, but if you ask the experts, they will give you a range or they will choose something within that range from a few years to a few decades. Some more pessimistic people think it could be a century, um. my own guess is it could be five to 20 years with a pretty high probability, like more than 50%, um, but that's not much, it's like inside clearly not, hopefully not my life and clearly my life's children and society need time to adapt. to these things so that if we move towards the human level of skills, I think a lot of questions need to be answered and hopefully before we get there, as we move down that trajectory in society, we try to deal with these changes, obviously people are talking about opportunities, some of which we were just mentioning, but TR money to be made, yes, there is no doubt about that, but there is the other side, the darker side, that people are putting their eyes on blank at the threats of potential AI, some are downright manic. about the dangers we face.

I'd like to go into some detail of some of the threats that you've certainly talked about, but just from 30,000 feet, yeah, where do you sit? I mean, are you worried? I'm worried about the whole spectrum of risks and I'm worried about the attitude we're taking towards those risks, so I'm worried that we're playing the sorcerer's apprentice, that we're playing with fire without understanding what the consequences might be. I mean, At least as a group we don't act like we understand the possible consequences. Maybe I'll use an example so you know that there are people who have been talking about geoengineering in the human atmosphere. to reduce greenhouse gases for sure, but we don't do it well, because we're not sure that we're not going to break the system and really that's the current situation in AI in the sense that we don't know how to build a system. of artificial intelligence that doesn't turn against humans or become a super powerful weapon in the hands of bad actors or be used to destroy our democracies, yeah, so I think you talked to some.

However, the leaders and like I guess this is naive and they will tell us why in detail, I mean, I can give my reasons why it's naive, but some leaders in the field say, look, if the time comes, we'll just We'll pull the plug, oh. Yeah, that's naive, yeah, so where do we do it? Why isn't that the answer? Why shouldn't we be allowed to sleep peacefully at night? Once an entity running on software decides to do something bad, it's not going to announce it. Hey, I've done it. Become a bad person, put me in jail, shut me down, right?

Whether it's some humans remote or it has lost control it will act preemptively so you can't turn it off and it's very easy if you have access to the internet you can do it and if you know how to program well enough to hack things , defeat some of our cybersecurity defenses, copy itself onto tons of computers. I mean, human hackers can do it. So if we have something comparable to human-level intelligence, that means we have machines that can program as well or better, probably better than our best programmers, so they will find a way to copy themselves in many places elsewhere.

So how is it done? turn it off right, so yeah, we should have, you know, off switches, but let's not count on that as the only defense in case something bad happens. Do you think that's part of the ability for us to not care or some of us, obviously a lot of people? we're worried, but do you think part of the ability of some of us to not worry about this is simply because it's so abstract? You have this thing called a computer living somewhere on this amorphous thing called the Internet, it just feels like a One step beyond the real threats that we can see is that what allows us to protect ourselves from the deep worry that we should have.

I think there are probably many possible reasons, um, the reason you're talking about is by the way. something that people cite as one of the reasons why most people don't pay much attention to the threat of climate change, it sure isn't in the here and now and how evolution has made us fear things we can see like a lion in front. of us, yeah, um, or a volcano that we can hear well and we can see and we can feel the heat or something, but if it's abstract like you say, it's harder to get excited about it, so I think that's one of the reasons which I think there are other reasons, if you're in the AI ​​business, well, you don't want to really hear what can go wrong because you're invested in the good side, let's say, or you expect to hear that yes, we will. keep it under control, we'll find a way, there are other reasons why I think many AI scientists are reluctant to consider that their work could be harming society, yes, it's a kind of psychological defense, right? we want to feel good about ourselves we, we don't want to feel guilty about something, did you go through a transformation of I don't know, denial, sure, yeah, sure, sure, for many years I wasreading about uh, some of the uh?

The concerns that people have been writing about for the last decade is not something new, but at least for the last decade I have been exposed to it, but I didn't take it very seriously. I was thinking, oh, it's in the far

future

and uh. Our current systems are too weak anyway. I just didn't pay much attention because I thought we're going to get a lot of benefits and you know, cure diseases and help us with the environment and education and everything, so let's go and harvest those. benefits um but of course I had to change my mind when the GPT chat came around I realized well this might come sooner than I thought and we're not ready yeah now your friend your colleague Yan Lun we had a conversation with him and me.

I'm sure you know better than I do that he's not as worried maybe as others because his point of view is at the end of the day, obviously I'm paraphrasing, but at the end of the day, if you have more good actors pushing. the frontiers of this research, then ultimately that is the best defense against bad actors, so you just go ahead and try to create the best AI systems you can to eliminate the harmful effects that can come from bad actors. actors, obviously. I think it's probably good to have good people pushing the boundaries, but where do you see it?

Well, I wish he was right, but I don't have any evidence that that's the case, because we're talking about the future of our societies and, you know, destabilizing democracies and potentially, you know, destroying humanity. I think we need to be more careful, so for example, the scenario you talked about assumes that if you have a battle between good AI and bad AI. that, you know, at least the defender has an advantage or is not worse off, yes, but that is not at all clear and, for example, in the context of biological weapons, experts think that the attacker has an advantage.

I give you an example of brotherhood, yes, could you have a laboratory working for six months on the development of a dangerous, lethal and very contagious virus, all of this silently without you knowing it, shouting to the world that we are doing this and then releasing it in many places at the same time? Maybe at the same time and now the Defenders have to fight quickly to find a cure, yes, and in the meantime people are dying, so yes, it will depend on the type of attacks and defense and I don't think we can put all our effort into it.

We shouldn't lay all our eggs, oh, it's like in human society, so in human society, if you have enough good guys, they can always win against the bad guy from a long time ago, but that's not necessarily true in general in the example of biological weapons, is clearly not true, yes, and so is your fear more for that type of example where some bad actors take advantage of advances in artificial intelligence or rather the example of climate change where you believe that You are going to clean the atmosphere but you ruin the world. in some unintended way and what would be, I mean, what is your worst fear in terms of the unintended harmful consequences of AI?

Well, I worry about all the things that can happen, but the worst, of course, is what people call loss of control, so maybe let me use an analogy to explain what loss of control is about. There are many ways in which that you can lose control but the one that scares me the most is the following is when the AI ​​because it has been programmed to maximize the rewards we give it the rewards that give it to it when it behaves well, this is how we train these systems at this moment, we train them like your cat or dog, giving them positive or negative rewards depending on their behavior, but there is a problem with that, first, they might have a different interpretation of what is right and what is wrong, so think about your cat and you are trying to train him not to get on the kitchen table and he lets you know to yell at him when you're in the kitchen and see him on the table but what?

You can understand that I shouldn't get on the table when the teacher is in the kitchen, that is my very different proposal, yes, then that type of imbalance called misalignment is already a little scary, yes, if it weren't a cat, but it was something more powerful but it's worse than that imagine it something is something more powerful like uh it's not a cat it's a grizzly bear and that's okay we know the grizzly bear could overpower us we're building we're going to build these agis We're going to be smart enough , so we are going to try to have some defenses, so we put the Bear in a cage, but right now we have no visibility on how we could build that cage that is guaranteed to keep the Bear in forever and in fact, everything we have tried has been defeated, so people make these jailbreak messages for example, which break all the defenses that companies working on AI have been able to discover, maybe one day we will discover how to build a really secure cage, but in this We don't know what that means at the moment.

It means that when the bear becomes smart enough or strong enough, it breaks the door, breaks the lock, hacks it, maybe knows it using cyber attack and comes out. and you know when he was in the cage you were training him by giving him fish when he behaved, same for the AI, right, you give him positive feedback, but now he can grab the fish from your hands, he doesn't do it even once. takes the reward and controls the mechanism by which he obtains the reward. He doesn't care what we want. He cares about making sure we maintain control over the fish, which maybe we don't want, so there's a conflict and he wants to. to make sure we never put him back in the cage, so he needs to control us or get rid of us, so what do we do?

I mean, obviously some people use these words railings, we put up railings and that's you. version of the jail, the cage that you were referring to, but that's enough, what we should do and how we should do it, okay, so there is no silver bullet, but here are some things that I have been advising governments . to do, including testimony in the US Senate in the summer, so we need those legal guardrails to make sure that the companies that are building these very powerful systems follow the best possible practices that we have in terms of security and at some point when we approach AGI if they can't prove to the public to the regulator that their system is secure enough then they shouldn't even build it.

We haven't gotten to that point yet, but I think this should be the strategy, but I mean that regulation. I mean, the response of some, of course, is certain, so the good actors will abide by that pronouncement that they shouldn't build it, which will only allow the bad actors to do it themselves and then where is that? It is not a concern, it is absolutely And that is why it is also necessary to have international treaties and it is also the reason why we should not even assume that the regulation and the treaty will be 100% efficient but they are going to reduce the number of bad incidents, so if something is criminally punished, there will be fewer people doing it, um and if most countries impose this type of thing, there will be fewer people doing it, fewer organizations, less, it will have to be like terrorist groups or rogue states, so we reduce the number of cases and then you have to prepare for the day when I don't know that North Korea does it anyway and we need to have our own good eyes, like Yan was saying, in order to help us defend ourselves, we need to think carefully. about that and before we do all this, I mean, before we build a good AI to defend ourselves, we need to make sure we know the recipe to do it in a way that doesn't, we don't lose control of it, so one of What What we have to do is a massive investment in AI safety research to figure out how we build a cage that the bear can't get out of so that the AI ​​can become our ally in case the regulations are not 100%. right to do it.

It sounds like a tall order when it comes to something so complex to be able to have that kind of foolproof system by which you can eventually lock it down. Well, yes, the question is: do you have a better idea? No, I do not have it. No, but I guess the question again becomes: Does this slow down the pace at which you do your own research? Do you think we should slow things down because we don't know where it's going or do you recognize the dangers and move forward, um, none of these really, so what I'm doing is trying to solve this control problem, like how do we build a safe cage?

I see and I think we should invest a lot more in that or be prepared to stop and, you know, slow down our industry, which of course, you know, we have a lot of good reasons for not doing it right. I mean, F*ck to death for the most part, right. I guess I mean there wasn't, no, no, it depends. No, I think some, so when I talk, I've been talking to several governments and people in government who have been working in Homeland Security understand it because they're used to thinking about the strange possibility that something really bad could happen. and try to put protections in place, yes, to minimize those risks, um, and yes, the response has been very different from different governments and I think the level of understanding of the threat is still something that is lacking in most governments, um, of course, the US government and the British governments have been very proactive in those directions, yes, but, you know, I see that other governments like to listen and not necessarily act yet, but do you think that Have AI safety concerns really had an overall impact on the rate of research?

Let's say in the United States or in Canada it has slowed things down at all, not at all, not at all, that's more what I expected, the answer would be, you don't know what, but the pace of research into the safety of the AIC has increased, Sure sure. I mean, it's not just me realizing, we have to do something about this, I mean, scientists have to do something about this, so I see more and more people willing to focus their energy on their science, their research on this kind of thing. of questions. Um, we, we really need to think about all of this as a collective decision-making problem, like going back to climate change again.

If we were collectively rational about it, it would be resolved like this, okay, just raise the carbon price around the world. the planet at a reasonable level and everyone becomes vegan, that's the other, well, maybe part of it you already know, and similarly, for AI there are solutions, but economic political forces, competition between companies, competition between countries, they are playing against STS, yes, just like I mean a Titan on the field that is working in that cage, where would you say we are in that effort? Do you feel confident that this is something feasible? I think to some extent yes, so I'm among a small group of researchers who I think we have a chance of finding provable security guarantees or at least asymptotically testable security guarantees, um, which would be, it would be much better than no guarantee, yeah what is the current situation um and Unfortunately I get the impression that mostly in the industry we are trying to take small steps to try to increase safety, but without really addressing the bigger problem of how to make the cage really safe , so the things that are happening now are good. but insufficient by far if we got to AGI too soon, so, putting aside concerns that run deep for a moment, when you think about and analyze the various AI applications that are already advancing rapidly, is there one or a collection that you look at?

It's very exciting because of what it can do for the world. Can you give us an example of some that excite you? I'm more worried than excited when I see progress now. I used to do it. be excited, uh, so color this threat, color your perspective, how deeply you look around you, the work that you and your colleagues have been doing and it fills you with a certain kind of fear, I guess, where it could go, um I mean, we. We're progressing faster on capabilities like making the bear bigger and smarter, yes, and how we build a better, safer cage, but when you think about it, you know I've read about several systems where you know the Doctors can have an enormous amount of cutting-edge research at their fingertips without you knowing how to read anything.

I mean, that sounds like the positive aspects of AI and we're just seeing the tip of the iceberg, yes, I think the potential benefits are immense, yes, and they've been with me for many decades, of course, right. I mean, I talked to Eric SCH not long ago and he was talking about how everyone is going to have their own AI assistant, this savant in their pocket. I think he described it in terms of the ability to just have a genie sitting on your shoulder, you know, 24 hours a day, uh, it's something that you imagine in the next few years will be as common as the cell phone.

I don't know about the timeline butyeah this is where we go uh and and where I mean is it's kind of exciting or again horror just colors everything in sort of dark gray tones and here's the problem so consider the risks as the magnitude of the risks in one by on the other hand, and the magnitude of the profits, on the other hand, sure, the problem is that they don't match. Okay you have a dollar and you are going to make a bet it will be $2 if everything goes well or you lose everything it is a good bet well I am a very conservative gambler and a conservative investor if this is your only bet you are not going to repeat That, you know that once you've lost everything, that's it, you can't invest.

You're absolutely right, so that's the kind of scenario we're in, where we could lose so much that even all the profits you can give me aren't worth it. Now what I do believe is that progress is complicated. we are achieving capabilities, so what I am writing is that the approach I want to take to try to solve the problem is to exploit the advances we are making in AI capabilities to build a more secure cage. One way to think about this is that if the AI ​​better understands what is right and what is wrong, then it will be less likely to do something wrong.

Let's say it's not the only concern, but that's an example to illustrate why having more capacity can help us with the, you know, reduce the damage that systems can create in the world right now. You lead. I compile it and correct me if I'm wrong, if I don't know the details, but I know an important Research Institute dedicated to AI in Montreal, how. How many people do you have? Generally speaking, you know, beneath you. I don't have anyone under me. I don't like that image, but there are about 100 researchers, mostly grant students, okay, and there are like 50 professors involved. who are residents at the Mila uh research center and 50 others who have access and our similar associate affiliates if you wantYes, it's a big powerhouse of machine learning here in Canada and in the world in terms of scientific impact and we're training many new students, of course, I'm sure it's something the world needs, but I asked for a specific reason if If maybe you've already done this, if you did a survey of all those people who are in an Institute that you are deeply connected to, Do the vast majority of them have the same perspective as you or are you an anomalous relative?

To the community, I'm part of a minority of people who are very concerned because there is a large silent majority, as is usual with a lot of these things, who simply haven't invested enough brain cycles into this particular question to have a firm opinion on it. one sense or another. the other right because because that's what you know, scientists are so focused on their particular problem and it's hard to move your focus to something broader like society, humanity, democracy, that little extra detail, right, yeah, no , exactly, but then, for example, do you do it? you spend time valuing or maybe that's the wrong word you spend time trying to get people to a place where they do think about this yes and it's effective there, for example, um maybe I don't know, I don't have metrics for that but um but I think I can move the needle a little bit in two ways given my position and my experience, one is on the scientific side of yes, progressing eye safety and the other is on the political side, in other words. make more citizens understand the risks and benefits and governments so we can collectively make better decisions.

So yeah, I'm talking to the media, I'm talking to the governments, and I hope we're doing a little bit of that now, you know, in the Best of everyone, thank you, thank you, this is it, are you Is the only research you're focused on now about AI safety or do you still have projects in the future that just push the theoretical, the limits of theoretical understanding? I have projects that were started before 2023, but many of them are actually connected, so I've been working on what's called probabilistic inference, i.e. how we can train new Nets to estimate complicated conditional probabilities that things actually are. that we need to sort in my opinion to get these kinds of probabilistic security guarantees and you know a lot of people.

We started this conversation by saying that we are trying to better understand human intelligence. I'm talking about a lot of people trying to quantify how this Inside our heads works are based on questions of basic knowledge and updating probabilities and trying to understand that you know what the most sensible decision is in the future. Do you see again in this current work an interaction between what's happening up here and how you're trying to do it? To bring it to light, yes, I'm still very inspired by human cognition, um, in the decisions that I'm making in my research, both in probabilistic inference work and in security work, yes, because the problems that we're dealing with resolving.

They are technically intractable in the sense that to do these things perfectly you would need an exponential amount of computing, but human brains do a good job, yeah, okay, so what are the tricks that your brain uses, especially the cognition of high level, the part that allows us? for example, as a scientist, considering multiple hypotheses simultaneously and discovering which ones might be good candidates. Yes, these skills are actually extremely useful in the security context. The reason is that, to be safe, you should look at the worst possible but plausible thing. scenario that could happen, surely it is a standard way, like a way of risk management, of doing things and, to do that, you need to be able to come up with those scenarios that are plausible, that are compatible with everything you know and also predict that something Something really bad would happen that you can't do it, yeah, that brings me to my final topic if you have enough patient to sit with us for a few more minutes, which is this question of conscience, right?

If you are talking about human beings. Intelligence and the human brain, yes you can, and last night I spent some time reading your article on AI and consciousness with a variety of authors where you reviewed this, you know, you know some of the interesting theories of attention schema awareness. Global theory. workspace and just trying to see how close these systems are to having qualities that align with those theories of how Consciousness works. Yes, I'd like to talk about that in a moment, but first, just to get it out there. Can you imagine that these systems could at some point have internal worlds that have experiences of the type we call conscious experience?

Yes, um, but in order to say something like this we will need to better understand what Consciousness really is. It means mechanically in the human brain, something we don't know well at the moment, so I've worked on some of those theories and I think there are plausible theories that are anchored in Neuroscience for Consciousness, at least in part. That's the most mysterious thing called subjective experience, like knowing what it feels like to see something or have a particular thought or emotion or whatever. I think that part can have a pretty simple mechanistic interpretation in terms of mathematics and if that kind of thing is true. so maybe it's not so mysterious anymore and how is that?

How sure are you? I mean, I've also spent some time going over integrated information theory, you know, Michael Graziano's attention schema. In the end they all left me feeling good, maybe it's some kind of model for what might be happening, but it never really answers the question of how a set of particles moving in one way or another can possibly provide an experience. internal. Have any of these illuminated that question or, if you're working with AI, have they illuminated that I'm talking about a different theory that's related to these and maybe complementary to some aspects of these, but that's more anchored in um as a new net interpretation of things? um with some dynamic system so I can briefly explain what it is um so some of the most mysterious properties of subjective experience is that it is ineffable in other words you can't easily translate it into words yeah um it's very rich what kind of connected to that same property is very rich and we can't we can't communicate it uh something is always missing when we try um it's very personal subjective like my experience is different from yours and it's fleeting so maybe you remember your experience from you know, 5 minutes ago or a time and it's actually the memory is not the same as what you experienced, it will be another experience, um, so it's something that happens in the moment, yes, it turns out you can get it. all these properties with a model that considers the evidence from Neuroscience that when you become conscious, the dynamics of the activity of neurons has a mathematical property called contractive that causes the activity to converge to some place called attractor and that place becomes in thought. that you are, you know that you have and you're getting closer to it, maybe you even move on to the next thought before you get there, but you've been getting closer to it and then you know another one and another one and that those attractors by sort of mathematical properties here form a finite and innumerable set, in other words, it is as if your brain is in a very high-dimensional continuous state, which is the activity of all the neurons, but at the same time it is between one of an innumerable set, like a discrete set as a sentence of possible places where it could be and therefore it has a dual symbolic nature in continuum, just as we have in the new modern networks, you know we have symbols but they are associated with vectors, in other words, what happens It's when you experience something you have. the complete and continuous high dimensional state of your brain, but what you communicate is what and also what goes into your memory is this very special state that is an attractor and that can be translated into symbols because it is discrete by nature, you know that could be. translated into a finite number of symbols and because we can only communicate these symbols I cannot communicate to you the full state of the brain and of course even to interpret that full state you also need to have my net weight M which is even more bigger than my neural network state, yeah, that's why it's ineffable because there's no way we can communicate that it's an overly large number like 10 to the power of 11 or something like that and um it's very rich because it's such a high dimensional thing um It's fleeting because it's happening at the moment you have that thought that you have this trajectory and the next time you approach the same thought from a different place, it's going to be a different experience, right?

And of course it's personal because it depends on the neural net weights in your brain, which are different than mine, so those symbols really have different associations and different meanings for you and me, so if that theory is correct , then the subjective experience is just a side effect. of a particular type of calculation that has meaning because those thoughts are useful in achieving, you know, reasoning and anything that we do with thoughts, um and the way the brain implemented those calculations is with this Dynamics Machinery and so on that we gives those feelings. I mean, and it sounds again, maybe I'm too quick, it sounds like a more rigorous mathematical version of intena schema theory where yes, and it's something that you could implement correctly, yes, so I mean, because we can't see the inner workings behind every thought. thoughts seem to float freely and give that ineffable quality that makes them feel incredibly mysterious, so I definitely feel like it's the right direction to go in, so there are social consequences to that kind of mechanical understanding of Consciousness. um the problem is that we humans associate Consciousness and subjective experience in particular with all kinds of things like intelligence, yeah, which is a little bit different and we also associate Consciousness with moral status, like you know you have rights, you have a right to exist, yes, um, we can.

Don't you know we can't turn it off, but if we have artificial intelligence systems that have similar mechanisms, then you could say that they have the mechanisms of Consciousness, they have subjective experience, they have all the attributes of that, then some people are going to Well, so they should be considered human beings, they should have rights and they should have them, basically we should not be allowed to turn them off and that is a dangerous slope that we do not understand enough and that could lead to these systems becoming more powerful than us, which is not the case of humans, since each human can be defeated by a group of other humans.

Artificial intelligence systems will be like a new species that could be smarter than us and I think we should be very careful beforehand. We make these kinds of moves and think about the consequences for ourselves if we take a risk with the future of humanity. I think we would like to wait and think very carefully about this. Which is thePossibility probably not? It's going to happen, but is there a chance for that to happen? And that's not a very popular way of saying it, but are we putting too much stock in the particular way we take in the sense that you know the vast majority of species that have ever lived have gone extinct?

So that's the natural course of events now here, in a sense, we would have given rise to these AI systems, not in the literal biological sense, but certainly in an intellectual and technological sense, so if they are the continuators of our species and they are more robust and more intelligent and can do things that this gray thing will never be able to do since it is limited in space and time and in computational capacity, is it so bad if that is the problem? I don't know, the problem is that they may be very different from us, for example, they may not care as much about others.

We are not always so affectionate as a species, although it is true, but we do it and my concern is to create something irreversible. for Humanity that we don't understand that could lead to something that is not as good as you make it out to be, yeah, um, if we build machines like we build them now that just try to maximize the reward, they are dumb in some ways and very intelligent in other ways that they are very non-human and I'm not sure that this is what we want for the evolution of intelligence and I'm not sure that if you ask 99% of humans if they think: Oh, are you okay?

We are going to replace you with a very unpopular perspective. Yeah, um, so if we go for democracy, I think that plan won't work. Yes, I totally agree with that. I completely agree with that. Let me ask one last question before I end here. You know, I think many people are more familiar now than they were a year ago with Oppenheimer's reaction after successfully building the bomb. You know, I've become a deadly destroyer of worlds based on an ancient sacred Sanskrit text to try to capture this feeling of having radically changed the world. world in a way that I wasn't so happy with because of where that kind of weaponry can go as you look at your own life's work, do you get the sense that, I mean, are you afraid that that's a place where can you find yourself?

Clearly I don't want to contribute to things that are going to be extremely destructive to our societies and human well-being. I don't feel like I've contributed that much, I mean. Compared to all the recognition and awards I've gotten, but I do feel a responsibility to do what I can to reduce harm of all kinds, including harm that's already happening, so there are some parallels, but I believe there is a scientific star system. There is something that bothers me a little, I think we need a little more humility in science, but everyone can do their part to make a better world, so we should all feel much more responsible, yes, in our choices as scientists, citizens or politicians, yes. so look, all of us thank you, I personally thank you for all the work that you're doing to try to make the world safer as we rapidly advance these new and exciting, but also scary on some level technologies, so thank you for spending . the time we're here today and I think it's a message that everyone should hear, so thank you very much for joining us, thank you for the great questions, thank you okay, thank you all for joining us in this conversation again, as always, please sign .

Subscribe to our newsletter Join our YouTube channel so you can receive alerts when these shows come out. We will have an ongoing series of these conversations on these critical issues of the day spanning the realm of science and technology and society as I just heard developments that have the ability to radically change the future, both for the worse and for the better. If you're in New York around May this year, join us for our live events if you're in Brisbane, Australia around March this year. We'll also be there doing some live events and again stay tuned for alerts when we release these new shows as we'll be posting a lot in the coming weeks again.

Thank you very much for joining us and signing out. Brian Green from New York at the World Science Festival