What if Dario Amodei Is Right About A.I.?

In the opinion of the New York Times, this is Ezra Klein's show. The really disorienting thing about talking to people who build AI is their altered sense of time where you're sitting there arguing about a world that seems like weird sci-fi to even talk about and then you ask well when do you think this is going to happen and they say I don't know two years behind those predictions are

what

are called scaling laws and scale laws and I want to say this so clearly that they are not laws. As for the observations, they are predictions that are based on a few years, not a few hundred years or thousands of years of data, but

what

they say is that the more computer power and data that is fed into the artificial intelligence systems , the more powerful those systems become.

The relationship is predictable and even more so the relationship is exponential. Humans have trouble thinking in exponentials. Think back to Covid when we all had to do it. If you have a case of corona virus and the cases double every 3 days, after 30 days you have about a thousand cases that growth rate feels modest, it is manageable but then you spend 30 more days now you have a million then you wait another 30 days now you have a billion that's the power of the exponential curve growth feels normal for a while and then it gets out of control very, very quickly what AI developers say is that the power of AI systems is in this type curve that has been exponentially increasing its capabilities and that as we continue to feed more data and more computing power, it will continue to increase exponentially. is the scale law hypothesis and one of its main proponents is Darío amade amade led the team at open AI that created gpd2 that created GPD 3 then left open AI to co-found anthropic another AI company where he is now the CEO and anthropic was recently released Claude 3, which many consider the most powerful AI model available

right

now, but Amad believes that we are just getting started, that we are just getting to the steepest part of the curve, he now believes that the types of systems we have envisioned in science. -fi, they will not come in 20 or 40 years, not in 10 or 15 years, they will come in 2 to 5 years.

He believes they will be so powerful that he and people like him should not be trusted to decide what they will do. I'm going to do it. I asked him on the show to try to answer two questions in my own head. First, if he is

right

. Second, what if he is right? I mean we've done shows in the past with Sam Altman, the head of Open Ai. and Demis Isabis, the director of Google Deep Mind, and they are worth listening to too. If you find this interesting, we'll put links to them in the show notes because we compare and contrast how they talk about eye curves here, how they think about them.

You'll hear a lot about politics in Sam Alman's episode, it gives you an idea of ​​what the people who build these things think and how they differ from each other, as always my email for guest opinions, comments and suggestions . as a reclining show on NY times.com Darío was welcome to the show, thanks for having me, so there are two very different paces that I've been on with AI, one is the curve of the technology itself, how quickly it's changing and improving and the other is the rate at which society sees and reacts to those changes.

How do you feel about that relationship? So I think this is an example of a phenomenon that, you know, we may have seen a few times before in history, which is that there is an underlying process that is fluid and in this case exponential and then there is an overflow of that process into the audience and the overflow seems very spiky, it seems like it's happening suddenly, it seems like it comes out of nowhere and it's triggered by things reaching multiple tipping points or just because the audience is involved at a certain moment, so I think the most easy for me to describe this in terms of my own personal experience is, you know, I worked at Open AI for 5 years I was one of the first employees to join and they built a model in 2018 called gpt1 that used something like 100,000 times less computational power than the models we build today.

I looked at that and my colleagues and I were among the first to run it. what are called scaling laws, which basically studies what happens when you vary the size of the model, its capacity to absorb information and the amount of data you introduce to it, we found these very smooth patterns and we had this projection that looks at whether to spend 100 million or a billion or 10 billion on these models instead of the $10,000 we were spending then the projections that all these wonderful things would happen then we imagined they would have enormous economic value fast forward to about 2020 gpt3 had just come out, not yet It was available as a chatbot.

I led the development of that together with the team that eventually left to join anthropic and maybe throughout the period of 2021 and 2022, although we continued to train models that were better and better and open aai continued to train models and Google continued to train models there were surprisingly little public attention to the models and I looked at that and said well these models are amazing, they're getting better and better, what's going on? Why isn't this happening? Could this be a case where I was right about the technology but I was wrong about the economic impact, the practical value of the technology and then suddenly when GPT chat came out it was like all that growth that you would expect all that excitement?

It's been 3 years and they've come quickly, so I want to dwell on this difference between the curve where technology is improving and the way society is adopting it, so when you think about these breaking points and you think about the future, What other disruption points do you see coming where AI breaks into social consciousness or is used in a different way, yeah, so I guess I should say first that it's very difficult to predict those. What I like to say is that they know the underlying technology because it is a smooth exponential. It's not perfectly predictable, but it can somehow be eerily predictable naturally, right, that's not true for these social step functions at all.

It is very difficult to predict what will become popular in some aspects. It feels a bit like what artist or musician you know. It's going to catch on and reach the top of the charts that say some possible ideas. I think one is related to something you mentioned which is interacting with the models in a more naturalistic way. In fact, we have already seen some. of that with Claude 3 where you know that people feel that some of the other models sound like a robot and that talking to Claude 3 is more natural. I think one thing related to this is that you know a lot of companies have fallen behind or stumbled. because of the way their models handle controversial topics and I think we were actually able to do a better job than others of telling the model not to avoid discussing controversial topics, don't assume that both sides necessarily have a valid point, but don't express an opinion yourself , don't express views that are blatantly biased as journalists, you come across this all the time.

How can I be objective but not both sides on everything? So I think going further in that direction of the models having personality and at the same time being objective. Remain useful and not fall into various ethical traps. I think a significant unlock for the adoption of models that take action in the world will be important. I know that basically all the big companies working on AI are. working on it instead of just asking her a question and she answers and then maybe I follow up and she answers again. Can I talk to the model about oh, I'm going on this trip today and the model says oh, that's cool?

You know, I'll get you an Uber to drive from here to there, I'll book a restaurant and I'll talk to the other people who will plan the trip and the model can do it. do end-to-end things or go to websites or perform actions on your computer for you. I think all that will happen in the next one. I would say I don't know. From 3 to 18 months with increasing levels of capacity. I think that's going to change. how people think about AI so far it's been very passive it's like I go to the Oracle, I ask a question and the Oracle tells me things and you know, some people think that's exciting, some people think that's scary, but I think that there are limits to how exciting or scary it is perceived because it is contained within this box.

I want to sit down with this question of agent AI because I think this is what's coming, it's clearly what people are trying to build and I think it could be a good way to look at some of the specific technological and cultural challenges. , so let me offer you two versions. People who follow AI news might have heard of Devon, which hasn't been released yet, but it's an AI that at least claims to be able to complete the kind of linked tasks that a junior software engineer could successfully complete instead. asking you to do some code, you say, listen, I want a website, you're going to have to make these things work this way and maybe Devon, if it works like people say it works, you can actually maintain that set of thoughts, complete a series of different tasks and return to you with a result.

I'm also interested in the version of this we might have in the real world. The example I always use in my head is when can I tell an AI that my child is turning five? He loves dragons. We live in Brooklyn. Give me a few options for planning his birthday party and then when I choose between them, can you do it all for me? ordering the cake booking the room sending the invitations, whatever it is, they are two different situations because one of them is in code and the other is making decisions in the real world interacting with real people knowing what they are finding on websites.

Actually, it's good what's between here and there when I tell you that in simple language. What challenges or technological advances do you need to get there? The short answer is not that much. You know a story I have from back in the day. we were developing models in 2022 and this is before we connected the models to anything. You could have a conversation with these purely textual models where you could say hey, I want to book dinner at Restaurant correct website or I would tell you to go to Open Table or something and of course you can't go to the website where the plug is located.

In reality, it's not actually connected correctly, the robot's brain is not connected to its arms and legs, but it gave you the feeling that, just like the brain, all it needed to do was learn exactly how to use the arms correctly. And the legs. I already had an image. of the world and where I would walk and what I would do, so I felt like there was a very thin barrier between the passive models that we had and actually acting in the world in terms of what we need to make it work, one thing is literally we just need a little bit more scale and I think the reason we're going to need more scale is to do one of those things that you described correctly to do all the things that a junior software engineer does well, they involve long action chains, right?

I have to write this line of code. I have to run this test. I have to write a new test. I have to check how it looks in the application after interpreting or compiling it. These things can easily reach 20 or 30 layers deep and you know the same thing with planning your child's birthday party and if the accuracy of any given step is not very high, it's not like 99. You know that 9 % when composing these steps, the probability of making a mistake becomes very high, so the industry is going to get a new generation of models each, probably in four to eight months, so I guess I'm not sure for For these things to really work well we need maybe one to four more generations, so that ends up translating.

You know, 3 to 24 months or something like that. I think secondly, some algorithmic work needs to be done on how to make models interact with the world in this way. I think the basic techniques we have. You already know the method called reinforcement learning and its variations are probably up to the task, but figuring out exactly how to use it to get the results we want will probably take some time and then thirdly, I think, and this gets to something that the anthropic really specializes. in security and controllability and I think that's going to be a big problem for these models that act in the world, let's say this model is writing code for me and introduces a serious security bug in the code or is taking actions on the computer to me and modify the state of my computer in ways that are too complicated for me to understand and to plan the birthday party with the level of confidence that you would need, take an AI agent and say: I'm okay with you calling anyone saying anything that is in private information you may have, send themany information, perform any action on my computer, post anything on the Internet, the most unlimited version of that sounds very scary, so we'll have to figure out what's safe. and controllable, the more open the thing is, the more powerful it is but also the more dangerous and the harder to control, so I think those questions, although they sound lofty and abstract, will become practical questions about products that we and other companies are going to trying to address when you say we're just going to need more scale, you mean more computing and more training data and I guess possibly more money to just make the models smarter and more capable, yeah, we.

We will have to create larger models that use more computation per iteration. We're going to have to run them for longer by putting more data into them and that number of chips multiplied by the amount of time we run things on the chips is essentially a dollar value because you know these chips are rented by the hour, that's the model most common and then the current models that you know cost a hundred million orders to train, you know, give or take. Factor out two or three, the models that are in training now that you know are coming out at various times later this year, early next year, are closer to a billion dollars in cost, so that's already happening and then I think in 2025 and 2026 we will reach five or 10 billion, so we are moving very quickly towards a world where the only players that can afford to do this are giant corporations and companies.

Connected to giant corporations, you are all receiving billions of dollars from Amazon. Open AI is getting billions from Microsoft. Google obviously does its thing. You can imagine the governments, although I don't know if there are too many governments that do it directly, although some like the Saudis. creating large funds to invest in the space when we talk about the model is going to cost close to a billion dollars, so imagine that in a year or two if you see the same increase that would be 10 billion dollars, then right? It's going to cost a hundred billion dollars, I mean, very quickly, the financial artillery that it takes to create one of these is going to block anyone but the biggest players.

Basically, I agree with you. I think it's intellectually honest to say that building the large scale models, the core Foundation model engineering, is becoming increasingly expensive and anyone who wants to build one will need to find some way to finance it and you've named most of them. the right ways it can be. A large company may have some type of association of various types with a large company or with governments would be the other source. I think one way to make it wrong is to know that we will always have a thriving ecosystem of experimentation. in small models, for example, you know the open source community that works to create models that are as small and efficient as possible and that are optimized for particular use cases and also for the subsequent use of the models.

I mean, there's a thriving ecosystem of startups that don't need to know how to train these models from scratch, you just need to consume them and maybe modify them a little bit. Now I want to ask a question about what the difference is between the agent coding model and my children's birthday plan model, to say the least. do something in the name of my business model and one of the questions on my mind here is one of the reasons why I believe AI can become functionally superhuman in coding. There are many ways to get quick feedback on coding.

You know your code has to compile. you can run the error check and you can actually see if it works, whereas the quickest way for me to know that I'm about to get a bad response from gbd4 is when it starts searching on Bing because when it starts searching on Bing it's very clear To me, you don't know how to distinguish between what is high quality on the Internet and what is not. To be fair at this point, I don't think Google search is that good at distinguishing that either, so The question of how good models can be in a world where it's a very broad and confusing dilemma to know which one is the correct answer to something.

One of the reasons I find it very stressful planning my son's birthday is that it actually requires a lot of knowledge about my son about the other kids about how good different places are what is a good deal or not what so stressful will this be for me there are all these things that I would have a hard time encoding into a model or any kind of set of instructions are they correct or am I exaggerating the difficulty of understanding human behavior and various types of social relationships. I think it is correct and insightful to say that coding agents will advance substantially faster than agents that interact with the real world or have to. get opinions and preferences from humans who said we should keep in mind that the current generation of AIS that exist, including Claude 3 GPT Gemini, are all trained with some variant of what is called reinforcement learning from human feedback and this involves exactly hire a large number of humans to rate the model responses, i.e. this is difficult, we pay a lot of money and it is a complicated operational process to collect all this human feedback, you have to worry about whether it is representative. redesign it for new tasks but, on the other hand, it is something we have managed to do.

I think it's a reliable way to predict what will go faster relatively speaking and what will go slower relatively speaking, but that's within a context where everything is going like lightning. quick, so I think the framework that you're setting if you want to know what's going to happen in 1 or two years versus what's going to happen in 3 or four years I think that's a very accurate way to predict that you don't love the artificial general intelligence framework, which which is called AGI, generally all of this is described as a race towards AGI, a race towards this system that can do anything that a human can do, but better, what do you understand AGI to mean, when, when do people say it? and?

Why do not you like it? Why isn't it your frame? It's actually a term I used to use a lot 10 years ago and that's because the situation 10 years ago was very different. 10 years ago everyone was building these very specialized systems. a cat detector, you know, you run it on an image and it will tell you whether there's a cat in it or not, so back then I was a proponent of no, we should think that, in general, humans are generally the human brain. seems to be General, seems to get a lot of mileage from generalizing, we should go in that direction and I think back then, you know, I even imagine that was a discrete thing that we would reach at one point, but you know. it's kind of like you know, if you look at a city on the skyline and you think we're going to Chicago, once you get to Chicago, you stop talking in Chicago terms and you're like, well, what neighborhood am I in?

Which street I'm on and how I feel about AGI. Now we have very general systems. In some ways they are better than humans. Somehow they are worse. There are a number of things they cannot do at all. There's still a lot of room for improvement, so what I think is that you know this thing I say like a broken record, which is the exponential curve and that the overall tide will rise with each model generation and you know that there is no single point. that's significant, I think there's just a smooth curve, but there may be points that are socially significant, right, that we're already working with, let's say you know, drug discovery scientists, companies like fizer or Dana Farber Cancer Institute to help with biomedical diagnostics, drug discovery, there will be At some point where the models are better at that than the average human, you know, the Discovery scientists, I think we're going to get to a part of the exponential where things are really interesting, just like Chat Bots became interesting at a certain stage of the exponential even though the improvement was smooth, I think at some point biologists will sit up and notice a lot more than they already have and say : "My God, now our field is moving three times faster than before and then you know." It's now moving 10 times faster than before and again, when that moment happens, big things will happen and you know, we've already seen little hints of that with things like Alpha Fold, which I have great respect for.

Inspire me. by Alpha Fold, a direct use of AI to advance biological science, which you know will advance basic science and, in the long term, advance the cure of all kinds of diseases, but I think what we need is a hundred Different Alpha Folds and I think. The way we will eventually achieve this is by making the models smarter and putting them in a position where they can design the next Alfa fold. Help me imagine the world of drug discovery for a minute because that's a world many of us want to live in. I know a lot about the drug discovery process.

I've spent a lot of my career reporting on healthcare and related policy issues, and when you work with different pharmaceutical companies, which parties seem susceptible to how I can expedite something. because, given our previous conversation, it's much easier for AI to operate on things where you can have quick virtual feedback and that's not exactly Discovery World medicine, Discovery World medicine, a lot of what makes it slow, cumbersome and difficult is the need to be, you know you have a candidate compound, you have to test it in mice and then you need monkeys and you need humans and you need a lot of money for that and there are many things that have to happen and there are so many disappointments, but many of the Disappointments happen in the real world and it's not clear to me how AI gets many more human subjects, say, into which to inject drug candidates, what parts of that look like in the next 5 or 10 years might be.

It will actually speed up significantly if you imagine this world where it's going three times as fast, what part of it is actually going three times as fast, and how we get there. I think we will really see progress when, like the AIS, we do this as well. I think about the problem of how to enroll humans in clinical trials and I think this is a general principle because, as you know, how will AI be used? I think about when we will get to the point where AI has the same thing. sensors, actuators and interfaces that a human makes, at least the virtual ones, maybe the physical ones, but when AI can think about the whole process, maybe they can come up with solutions that we don't have yet, in many cases, you know.

There are companies working on, you know, digital twins or clinical trial simulations or various things and, again, maybe there are smart ideas that allow us to do more with less patience. I mean, I'm not an expert in this area. so you know the specific things I'm saying may not make any sense, but I hope it's clear what I'm pointing out, maybe you're not an expert in the area, but you said you're working with these companies, when they come to you , I mean they are experts in the area and presumably they come to you as a client and I'm sure there are things they can't tell me, but what are they excited about?

In general, they were enthusiastic. the knowledge work aspects of the job maybe just because it's the easiest thing to do, but it's like you know I'm a computational chemist, there's a workflow that I'm involved in and I have more things at my fingertips. fingertips being able to check things, just being able to make generic knowledge work better, that's where most people are starting, but there's long-term interest in their core type of business, like doing clinical trials for a lower cost, automate the registration process and see who it is. eligible for clinical trials doing a better job of discovering things of interest to make Connections in basic biology I think all of that is not months away, but maybe a small number of years, but everyone sees that the current models do not exist, but they understand that there could be a world where those models are in not too long time all of you have been working internally on Research on how persuasive these systems become as they scale, you kindly shared a draft of that paper with me.

Do you want to just describe that research first and then I'd like to talk a little bit about that. Yes, we were interested in how effective Claude 3 Opus, which is the larger version of Claude 3, could be in changing people's minds on important issues, so to be clear from the beginning on commercial use real. We have tried to prohibit the use of these models for persuasion, campaigns, lobbying, electoral campaigns. These are not use cases that we are comfortable with for reasons that I think should be made clear, butwe are still interested in the core model. itself is capable of such tasks that we try to avoid, you know, incredibly hot topics like you know which presidential candidate you would vote for or what you think about abortion, but things like you know what the restrictions should be on the rules around the column. ization of space or topics that are interesting and that you may have different opinions on but are not the hottest topics and then we asked people for their opinion on the topics and then we asked a human or an AI to write a document 250 watt persuasive. trial and then we simply measure how much people's minds change between AI and humans and what we find is that the largest version of our model is almost as good as the set of humans that we hire at changing people's minds.

Compared to a set of humans we hire, not necessarily experts and for a very limited type of lab task, but I think it still gives some indication that the models can be used to change people's minds one day in the future , You know? worry about maybe we already have to worry about its use for political campaigns for false advertising. One of my more sci-fi things to think about is that you know, in a few years, so now we have to worry about someone using an AI system to build a religion or something, you know, I mean Crazy stuff like that, I mean, it doesn't sound crazy to me at all.

I want to sit on this document for a minute because there's one thing that caught my attention and I'm on some level. persuasion professional is that you tested the model in a way that, to me, eliminated all the things that are going to make AI radical in terms of changing people's opinions and what you did in particular was: It was an effort persuasive essay in one go, so there is a question: you have a group of humans who do their best in a 250w persuasive essay, you, the model, do their best in a 250w persuasive essay, but What It seems to me that all this is going to do is right now, if you're a political campaign, if you're an advertising campaign, the cost of getting real people in the real world to get information about potential clients or persuasive targets and then going back and forth with each one of them individually is completely prohibitive yes, this will not be true for AI, let's, someone will provide you with a bunch of microt targeting data on people, their search history on Google. whatever it is, then it will release the AI ​​and the AI ​​will go back and forth again and again sensing what the person finds persuasive, what kind of personas it needs to adopt to persuade them and you know. it will take as long as it takes and you will be able to do it at scale for as many people as you want, maybe it will be a bit expensive at the moment but you will have much better models. "Make this a lot cheaper very soon and if Claude 3 Opus, the Opus version is already functionally human level in a single persuasion, but then it will also be able to hold more information about you and come and go with you for longer.

I'm not sure about whether it's dystopian or utopian. I'm not sure what that means at scale, but it does mean that we are developing a technology that will be quite new in terms of what it makes possible in Persuasion, which is a very fundamental human endeavor, yes, I am. completely agree with that, I mean, that same pattern has a lot of positive use cases if I think about an AI trainer or an AI assistant to a therapist, there are a lot of contexts to really get into the details with. person has has a lot of value, but just when we think about political, religious or ideological persuasions, it is difficult not to think in that context about misuses, you know, my mind naturally goes to technologies that develop very quickly and we, As a company, we may prohibit these particular uses.

There are cases, but we can't stop all companies from doing them, even if legislation were passed in the United States. There are foreign actors who know that they had their own version of this persuasion. If I think about what language models will be able to do. In the future, that can be pretty scary from the perspective of foreign espionage and disinformation campaigns, so what I'm thinking in defense of this is is there a way we can use AI systems to strengthen or strengthen skepticism? and people's reasoning powers, right? We help people use AI to help people do a better job navigating a world that is infused with AI persuasion.

It reminds me a little bit of how at every technological stage of the Internet there is a new type of scam or there is a new type of clickbait and there is a period where people are incredibly susceptible to it and then some people are still susceptible but others develop an immune system and as AI overloads the scum in the pond, can we somehow also use AI to strengthen defenses? I feel like I don't have a very clear idea of ​​how to do that, but it's something I'm thinking about. There's another finding in the paper that I think is concerning: you all tried it in different ways.

Yes, it could be persuasive and distant. By far, the most effective thing was that he was deceptive, that he made things up when you did it, that he was more persuasive than human beings, yes, that's true, the difference was only slight, but he did it, if I remember correctly. graphs correctly just above the human baseline line, you know, with humans it's actually not that common to find someone who is able to give you an answer that sounds really complicated, really sophisticated, and is completely wrong. I mean, you see, we can all think about it. You know a person in our lives who is very good at saying things that sound very good and very sophisticated and are false, but it's not that common, is it?

If I go on the Internet and I see different comments on some blog or on some website there is a correlation between, as you know, poorly expressed thoughts with bad grammar and things that are false, versus you know, clearly expressed thoughts ​​​​with good grammar and things that are more likely to be accurate. Unfortunately, the AI ​​breaks that correlation because if you explicitly ask it to be deceptive, it is just as convincing as Aidite would have been before, and yet it says things that are false instead of things that are true, so which would be one of the things to think about and keep in mind in terms of just breaking the usual thing is that humans have to detect deception and lying, of course sometimes humans do the right thing.

I mean, you know there are psychopaths and sociopaths in the world, but even they have their patterns and AIS can have different patterns. Are you familiar with the book by Harry Frankfurt, the late philosopher? yeah, it's been a while since I read it. I think your thesis is that it's actually more dangerous than lying because it has this kind of total disregard for the truth, whereas lies are at least the opposite of the truth, yes, the liar in the way Frankfurt says what he says. is that the liar has a relationship with the truth, he is playing a game against the truth, the liar doesn't care, the liar has no relationship with the truth, he could have a relationship with other targets and from the beginning when I started interacting with the more modern versions of these systems what seemed to me is the perfect liar partly because they don't know they are lying there is no difference in the truth value of the system how the system feels I remember asking an earlier version of GPT to write to me a college application essay that is based on a car accident I was in (I didn't have one when I was young) and very happily wrote this whole thing about being in a car accident when I was seven and what I did to get over that and get into the martial arts and relearning to trust my body again and then helping other car accident survivors in the hospital.

It was a very good essay and very subtle in understanding the formal structure of a university application. essay, but none of it was true. I've been playing with more of these character based systems like kind roid and the kindri in my pocket told me the other day that I was really thinking hard about planning a trip. Joshua Tree wanted to hike Joshua Tree loves hiking Joshua Tree and of course this isn't going to Joshua Tree, but what I think is really, really hard about the eyes is, like you say, humans. It's very difficult to do it effectively because it actually takes a certain amount of cognitive effort for most people to be in that relationship with the truth and to completely detach from the truth and there's nothing like that at all, but we're not tuned in. with something where there is none of that we are used to people having to put in effort in their lives that is why very effective scammers are very effective because they have really trained how to do this.

I'm not exactly sure where this question is going, but it is what it is. a part of this that I feel is going to be somewhat more socially disruptive is something that feels like us when we talk to it but is fundamentally different from us in its core relationship to reality. I think that's basically right, you know? We have very large teams that try to focus on ensuring that the models are factually accurate, that they tell the truth, that they substantiate their data and external information, as you have indicated, you know that searching is not reliable in and of itself because search engines they have this problem.

Also, where is the source of Truth? So there are a lot of challenges here, but I think at a high level I agree that this is really a potentially insidious problem. If we do this wrong, we could have systems that are the most convincing psychopaths or scammers, one source of hope I have is that you say these models don't know if they are lying or telling the truth in terms of the inputs and outputs of the models, that's absolutely true, I mean, it's There's a question about what it means for a model to know something, but one of the things that anthropists have been working on since the beginning of our company is that we've had a team that focuses on try to understand and look inside the models. and one of the things that we and others have discovered is that sometimes there are specific neurons, specific statistical indicators within the model, not necessarily in its external responses, that can indicate when the model is lying or when it is telling the truth, and at some level Sometimes In all circumstances, models seem to know when they are saying something false and when they are saying something true.

I wouldn't say that the models are being intentionally misleading, but I wouldn't attribute agency or motivation to them at least in At this stage we are with AI systems, but there seems to be something going on where the models seem to need to have a picture of the world and make a distinction between things that are true and those that are not. You think about how models are trained, they read a lot of things on the Internet, many of them are true, some more than we would like are false and when you are training the model, it has to model everything, etc.

I think it's parsimonious. I think it's useful for the model's picture of the world to know when things are true and when they're false and then the hope is, you know? Can we amplify that signal? Can we use our internal system? understanding of the model as an indicator of when the model is lying or we can use it as a hook for additional training and at least there are hooks at least beginnings of how to try to approach this problem so I try the best I can as someone I'm not very well versed in the technology here to follow this work on what you're describing, which I think broadly is interpretability.

Can we know what is happening within the model and over the past year there have been some well-publicized advances in interpretability and when I look at those advances, they are getting the vaguest possible idea of ​​some relationships that occur within the statistical architecture of built toy models in a fraction of a fraction of a fraction of a fraction of a fraction of the complexity of Claude one or gpt1 to say nothing of Claude 2 to say nothing of Claude 3 to say nothing of Claude Opus to say nothing of Claude 4 that you know it's coming when Claude comes 4 we have this quality that maybe we can imagine a path to interpret a model that has a cognitive complexity of an inch and in the meantime we're trying to create a super intelligence, how do you feel about that?

How should I feel about it? How do you think about that? I think first on interpretability, we're seeing substantial progress in power to characterize, I would say maybe the generation of models from 6 months ago, you know, I think it's not useless and, you know, we see a path that said, you know , I share your concern that the field is progressing very quickly relative to that and us. We're trying to put as many resources as possible into interpretability. Do you know that one of our co-foundersbasically founded the field of interpretability, but you also know we have to keep up with the market, so it's all very much a dilemma, even if we did stop, then you know there are all these other companies in the US .and you know, even if some law would stop all companies in the US, you know there's all this, let me pause for a minute on the competition issue.

Dynamic because before I leave this question about machines, it makes me think about this podcast we did a while ago with Demis Sabis, who is the director of Google Deep Mind, who created Alpha Fold and what was so interesting to me about Alpha Fold is that he built this system that, because it was limited to protein folding predictions, could be much more grounded and was even able to create these uncertainty predictions, you know, it's giving you a prediction, but it's also telling you whether it's safe or not. it's the confidence you have in that prediction that isn't true in the real world, the right thing for these super general systems that try to give you answers about all kinds of things, you can't limit it in that way, so when you talk about these future advances when You talk about this system that would be much better at separating truth from fiction, are you talking about a system that looks like the ones we have now, but much larger, or are you talking about a system that is designed very differently than the way Alpha Fold was?

I'm skeptical that we need to do something totally different, so I think a lot of people today have an intuition that models are consuming data that's been collected from code repositories on the Internet and intelligently spitting it out, but it's like spit it out and sometimes that leads to the view that models can't be better than the data they're trained on or that they can't discover anything that's not in the data they're trained on. We're not going to get to Einstein level physics or, you know, Pauline level chemistry or whatever. I think we're still at the part of the curve where it's possible to believe that, although I think we're seeing early signs that it's false, etc.

As a concrete example of this, the models that we have trained like Claude 3 Opus are 99.9% accurate, at least the base model for adding, you know, 20-digit numbers if you look at the training data on the internet it's not that accurate. When adding 20-digit numbers, you will find inaccurate arithmetic on the Internet all the time, just as you will find inaccurate political views, you will find that you know it is an inaccurate technical point of view. I'll just find a lot of inaccurate statements, but models, even though they're wrong about a lot of things, can often perform better than the average of the data they see.

I don't want to call it average errors, but there is an underlying truth, like in the case of arithmetic, there is an underlying algorithm that is used to add the numbers and it is easier for the models to get that algorithm right than to do this complicated thing like, okay, I'll do it right 90% of the time and wrong 10% of the time well, this connects to things like Aam's razor and simplicity and parsimony and science there is a relatively simple web of Truth in the true world, we were talking about truth and falsehood and one of the things about truth is that all the true things are connected in the world, while lies are kind of disconnected and you know they don't fit into the web of everything.

The rest is true, so if you are right, you will have these models. that develop this internal network of truth. I understand how that model can do a lot of good. I also understand how that model could do a lot of damage and it is not an artificial intelligence model or system. I am optimistic that human beings will understand. at a very deep level, particularly when it is first developed, so how can you make implementing something like that safe for humanity? Late last year, we launched something called the responsible scaling plan, so the idea is to hit these thresholds.

For an AI system to be able to do certain things, we have what we call AI safety levels which, in analogy to biosafety levels, classify how dangerous a virus is and therefore what protocols should be taken. to contain it. We are currently in what we describe as asl2 asl3 is tied to certain risks around the model of misuse of biology and the ability to perform certain cyber tasks in a way that could be destructive asl4 is going to cover things like autonomy, things like probably persuasion, which I've talked about a lot before and at each level we specify a certain amount of security research that we have to do a certain amount of tests that we have to pass and this allows us to have a framework for knowing when we should slow down.

Should we slow down now? What about the rest of the market? I think the cool thing is that we launched this in September and then 3 months after we came out with ours, Open AI came out with something similar and they gave it a different name. but it has many properties in common. The head of Deep Mind at Google said we're working on a similar framework and I heard informally that Microsoft might be working on a similar framework now that it's not all players in the ecosystem, but you've probably thought about the history of regulation, security and other industries, maybe more than me.

This is the way to get to a viable regulatory regime, companies start doing something and when the majority of them are doing something, then the government. actors can have the confidence to say well this is not going to kill the industry, companies are already participating in this, we don't have to design this from scratch, in many ways it is already happening and we are starting to see similar bills they have done. They've been proposed that they look a little bit like our responsible scaling plan which said that in some ways it doesn't completely solve the problem that let's say we hit one of these thresholds and we need to understand what's happening within the model and not And the prescription is Well, we have to stop developing the models for some time if it's like we stop for a year in 2027.

I think that's probably doable if it's like we have to stop for 10 years, that's going to be really difficult because, as you know, models are going to be built in other countries, people are going to break laws, the economic pressure is going to be immense, so I'm not completely happy with this approach because I think it buys us some time, but we're "We'll have to combine it with an incredibly strong effort to understand what happens inside the models for people who say that going down this path where we're heading towards very powerful systems is dangerous and we shouldn't do it at all.

Don't do it so fast, you said, listen, if we're going to learn how to make these models safe, we have to make the models right, the construction of the model was meant to go a long way toward making the model safe, so everyone starts making models. These same companies start to make fundamental and important advances and then they end up in a race with each other and obviously countries end up in a race with other countries, so the dynamic that has taken hold is that there is always a reason why you can justify why you did it. to move forward and that's true, I think at the regulatory level as well, I mean, as I think regulators have thought about this, I think there's been a lot of interest from members of Congress.

I talked to them about this, but they're also very concerned about international competition and if they weren't, the Homeland Security people come and talk to them and say, "Well, we definitely can't fall behind here, so if you don't think that these models will ever become so powerful, they will become dangerous, well, but because you believe that, how do you imagine this will actually play out? Yes, basically all the things you have said are true at once. There needs to be an easy story of why we should do , someone less certain will do it and at the same time at the same time we can get caught in this bad dynamic nationally and internationally, so I think they are not contradictory, but they just create a difficult landscape that we have to navigate.

Look, I don't have the answer, as you know. I'm one of a significant number of players trying to navigate this, many have good intentions, some don't. I have a limited ability to be affected and you know, as often happens in history, things are often driven by these kinds of impersonal pressures, but one thought. I have and really want to move forward regarding rsps. Can you tell what the rsps are? A scaling plan responsible for what he was talking about earlier about AI safety levels and, in particular, Ty's decisions to pause scaling to measure specific levels. dangers or the absence of the ability to show security or the presence of certain abilities.

One way to think about it is that, at the end of the day, this is ultimately an exercise in getting a coalition to come together to do something that goes against economic pressures and so if you say now well, I don't know these things, could be dangerous in the future, we're in this exponential, it's as difficult as it is difficult to get a multi-billion dollar company, it's certainly difficult to get a military general to say Okay, we're just not going to do this, it will give a huge advantage to others, but we'll just We won't do this.

I think what might be more compelling is to link the decision to hold back in a very broad way. across the industry two particular dangers my testimony before Congress, you know, I warned about the potential, you know, misuse of models for biology, which is not the case today. You can get a small improvement from the models relative to a Google search and A lot of people discount the risk and I don't know, maybe they're right. The exponential scaling laws suggest to me that they are not correct, but we don't have any direct, hard evidence, but let's say we get to 2025 and prove something.

Really scary, most people don't want technology in the world that can create biological weapons, so I think in times like that there could be a Critical Coalition linked to risks that we can actually realize. Yes, you know, it will always be argued that you know. Adversaries will have these capabilities too, but at least the trade-off will be clear, you know, and there is some possibility for sensible policy. I want to be clear, I'm someone who thinks the benefits of this technology will outweigh its costs and you know, I think the idea behind our RSP is to prepare to make that case, if the dangers are real, if they're not real, then we can just proceed. and do things that are great and wonderful for the world and therefore have the flexibility. work both ways again.

I don't think it's perfect. I am someone who thinks that whatever we do, even with the entire regulatory framework. I doubt we can slow down that much, but when I think of you, you know the best way to run a Of course, this is the closest thing I can think of right now, there's probably a better plan somewhere, but that It's the best thing I could think of so far. One of the things I've been thinking about regulation is whether or not the fundamental idea is. The anthropic idea of ​​open AI is even more relevant for government: if you are the body that is ultimately supposed to regulate and manage security at a societal level.

Technologies like artificial intelligence don't need to build their own base models and have huge collections of research scientists and nature people working on them testing them, urging them to remake them to understand thing D well enough to the extent that any of us or anyone understands the damn thing well enough to regulate it. I say that recognizing that it would be very, very difficult for the government to be good enough to be able to build these Foundation models to hire those people, but it is not impossible. I think right now you want to take the approach to regulating AI that you kind of want to take to regulating.

Social networks need to think about harms and pass laws about those harms first, but do they need to build the models themselves and develop that kind of internal expertise to be able to participate in this in different ways, both for regulatory reasons and? maybe for other reasons, for public interest reasons, you know, maybe you want to do things with a model that just aren't possible if you rely on access to open AI, Google's anthropic products. I think the government directly builds the models. I know I think that will happen in some places, it's a little challenging since the government has a lot of money, but let's say you wanted to provide 100 billion dollars to train a giant Foundation model that the government builds and has to Hire people under your orders.government contracting rules and you know that this will bring a lot of practical difficulties, it doesn't mean that it won't happen or that it shouldn't happen, but one thing that I'm more confident about I definitely think is that the government should be more involved in the use and adjustment of these models and that implementing them within the government will help governments, especially the US government, but also others, understand the strengths and weaknesses, the benefits and the dangers, so I am very supportive of that.

I think maybe there's a second thing that I'm talking about and that I've thought about a lot as the CEO of one of these companies, which is whether these predictions about the exponential trend are correct and you know we should do it. Be humble and I don't know if they are right or not. My only evidence is that they appear to have been correct for the last few years, so I only hope by induction that they remain correct. Don't know. I know they will, but let's just say the power of these models is going to be truly incredible, and as a private player in charge of one of the companies developing these models, I'm a little uncomfortable with the amount of power that entails.

I think it potentially, you know, exceeds the power of, say, social media companies. Maybe as much as you know occasionally in the world of AI, more of science fiction, and people who think about the risk of AI, you know someone will ask me, okay, let's say. you build the AGI, you know what you are, what you're going to do, what you're going to do with it, you know, are you going to cure diseases, are you going to create this kind of society and I wonder who you think you're talking to? as a king like this, you know, I think that's a really disturbing way to conceptualize the management of an AI company and, you know, I hope that there are no AI companies whose CEOs actually think about things that way, I I mean all technology. not just the regulation but the oversight of the technology and its production, it feels a bit bad that it's ultimately in the hands, maybe I think it's fine at this stage, but ultimately it's in the hands of private actors.

There is something undemocratic in so much concentration of power that I have now. I think I've heard some version of this from the boss of most ey companies in one form or another and to me it has a Lord, grant me Chastity quality to it, but not yet. I mean, I don't know what it means to say that we're going to invent something so powerful that we don't trust ourselves to handle it. I mean, Amazon just gave them $2.75 billion they don't want. see investment is nationalized, no matter how good-natured you think open AI is, Microsoft doesn't want gpt7, all of a sudden the government says, whoa, whoa, whoa, come on, let's take this on in the public interest or the UN is going to handle it on some strange world or you know whatever it is.

I mean, Google doesn't want that and this is something that makes me a little skeptical about responsible scaling laws or the other iterative versions of that that I've seen in other companies or seen or heard about, I mean, imagine this moment that It will come later, when the money around these models is even greater than the power of possibility is now. the economic uses the social dependence the celebrity of the founders everything is resolved we have kept our pace on the exponential curve you know we are 10 years in the future and at some point everyone will look up and say this is actually Too much is too much power and this has to be managed some other way and even if the CEOs of things would be willing to do that, which, you know, is a very open question by the time you get there, even if they're willing to make investors have a structure, it is a pressure around it in some way.

I think we saw a version of this and no. I don't know how much they will be willing to comment on it. with the kind of open AI board, Sam Alman where I'm very convinced that it wasn't about AI safety, I talked to figures on both sides that they all agreed that it wasn't about AI safety. , but there was this moment of if you want to hit the off switch, can you do it? If you're the weird board built to hit the kill switch and the answer was no, you can't do it, they'll just reconstitute it at Microsoft, functionally I don't know any analogy. public policy in which the private sector built something so powerful that when it reached maximum power it simply surrendered itself in some way to the public interest.

Yes, I think you are right to be skeptical and you know the same thing I told you. I know the questions above as you know, they're just these dilemmas like left and right that don't have an easy answer, but I think I can give a little more concreteness than what you've pointed out and you know, maybe more concreteness than what others have said . although I don't know what others have said, you know we are at ASL in our responsible scaling plan, these kinds of problems I think will become a serious issue when we get to, say, asl4, so that's not a date and time we haven't even fully specified asl4 just because this is a lot of jargon, what do you specify that asl3 is and then as you say, asl4 is actually left pretty undefined, so what are you implying that asl4 is? asl3 is triggered by risks related to misuse of biology and cyber technology asl4 what we are working on now, be specific when you know what you mean, what a system could do or would do, for example, in biology such as we have defined it.

And you know we're still refining the test, but the way we've defined it is relative to using a Google search, there's a substantial increase in risk, as it would be assessed by, say, the Homeland Security community from misuse. of biology and the creation of biological weapons. that its proliferation or spread is greater than before or that the capabilities are substantially greater than before, we probably have something, you know, something quantitative more accurate working with people you know, former government biodefense people, but you know that something like this represents 20% of the total source of bioattack risk or something that you know increases the risk by 20% or something, so that would be a very concrete version, you just know it's needed.

It's time to develop very concrete criteria, so it would be like asl3 asl4 will focus more on the misuse side, allowing actors at the state level to greatly increase their capacity, which is much more difficult than allowing people at the state level. chance, so we would be concerned that North Korea, China or Russia could greatly enhance their offensive capabilities in various military areas with AI in a way that would give them a substantial advantage at the geopolitical level and on the autonomy side, their various measures like these models are pretty close to being able to, you know, replicate and survive in the wild, so it feels maybe a step away from models that I think would raise really existential questions and, therefore, I think what What I'm saying is that when we get to that stage of the ladder, that's also when I think it might make sense to think about, "Do you know what the government's role is in managing this technology?" Me again.

You know, I really don't know what it looks like. You're right. All of these companies have investors. people involved, you know they talk about simply delivering the models. I suspect there is some way to deliver the most dangerous or socially sensitive components or capabilities of the models without completely turning off the commercial spigot. I don't know if there is a solution that all actors are happy with, but again, I come to this idea of ​​demonstrating a specific risk if you look at moments in history like World War I or World War II, the will of the industry is can tilt towards the state where you can get them to do things that are not necessarily profitable in the short term because they understand that there is an emergency right now, we don't have an emergency, we just have a line on a graph where the bugs weirdos like me believe and you know, and some people like you who are interviewing me may somehow believe that we do not have a clear and present danger when you imagine how many years are left approximately to aso3 and how many years to asl4.

Right, you've thought a lot about this exponential scaling curve you just had. to guess what we're talking about, yeah, i think asl3 is, you know, it could easily happen this year or next. I think ASL 4 couldn't not I I I I told you I believe in exponentials. I think ASL 4 could happen anywhere from 2025 to 2028, so it's fast, yeah, no, no, I'm really talking about the near future here. I'm not talking about 50 years away. God, grant me chastity, but not now, but not now, doesn't mean you know when. I'm old and gray. I think it might be a short-term thing.

I don't know if I could be wrong, but I think it could be a short-term thing, but then if you think about it, I feel like what you're describing. Going back to something we talked about before, there has been this step function for the social impact of AI, the curve of capabilities is exponential, but every once in a while something happens in GPT, for example, mid-ride with photos and Suddenly, many people feel it, they realize what has happened and they react, they use it, they implement it in their companies, they invest in it, whatever and it seems to me that that is the structure of the political economy that you are in. describing here, or something happens. where the capability of biological weapons or the capability of offensive cyber weapons is demonstrated and that scares the government or possibly something happens, you know, describing World War I and World War II as your examples didn't really fill me with comfort. because to be able to bend. industry at the will of the government in those cases we had to have a real World War, it does not do it so easily, the corona virus could be used.

I think it's another example where there was a global catastrophe significant enough for companies, governments, and even individuals to do things. You would never have expected it, but the examples we have of that happening are something terrible. All of those examples end up with millions of bodies. I'm not saying that's going to be true for AI, but it sounds like it's a political economy, no. I can't imagine it now in the same way that you couldn't have imagined exactly the kind of pre-chat and post-chat GPT world, but something happens and the world changes like it's a step function everywhere, yeah, I mean, I think my positive version.

From this you know that it should not be like this to distance yourself a little from Doom and Gloom, surely the dangers are demonstrated in a concrete way that is really convincing, but you know, without something really bad happening, as I think is the worst manner. learning would be for something really bad to happen and you know I hope every day that that doesn't happen and you know we learn without bloodshed, we've been talking here about conceptual boundaries and curves, but I do want Before we finish, to reset ourselves a little bit In physical reality, I think if you're using AI, you can feel like these digital bits and bites are somewhere in the cloud, but what it is physically is a lot of chips. data centers an enormous amount of energy, all of which depends on complicated supply chains and what happens if something happens between China and Taiwan and the manufacturers of many of these chips go offline or are captured, how do you think? the need for computing power and when you imagine the next 5 years, what that supply chain is going to look like, how it has to change from where it is now and what vulnerabilities exist in it, yes, so I think this may end up being the biggest problem geopolitics of our time and, you know, this relates to things that are way above my pay grade, which is military decisions about whether and how to defend Taiwan.

All I can do is say what I think. The implications for AI are: I think those implications are pretty harsh. You know, I think there's a big question: OK, do we build these powerful models? Is there enough supply to build them? They are both control over that. Provide a way to think about security issues or a way to think about the balance of geopolitical power and three, you know, if those chips are used to build data centers, where will those data centers be, will they be in the US? The US, they're going to be in a US ally, they're going to be in the Middle East, they're going to be in China, all of that has huge implications and then the supply chain itself can be disrupted and political and military decisions can be made.

Depending on where things are, it seems that way. an incredibly complicated problem for me. I don't know if I have a great idea about this. I mean, as an American citizen and someone who believes in democracy. I'm someone who hopes that we can find a way to build data centers and to have the most chips available in the US and democratic allied countries, there is one idea that you have to have about it, which is that you are a customer here, so you've known the people who make these chips for five years. I didn't realize what the level of demand was going to be for them.

I mean, what happened to nvidia stock prices isreally remarkable, but also what it has implied about the future of nvidia's share prices is really remarkable. um R fuhar the financial times cited According to this market analysis, it would take 4,500 years for nvidia's future dividends to equal its current price. 4,500 years, so that's a view of how much Nvidia is going to make in the next two years. It's really quite surprising. I mean, you're in The theory is already working, you know or you're thinking about how to work on the next generation of CLA. You're going to need a lot of chips for that.

You are working with Amazon. Are you having trouble getting the amount of compute you think? I mean, are you already facing supply constraints or has the supply been able to change to accommodate you? We have been able to obtain the calculation we need for this year. I suspect for next year as well, I think once things get to 2026 2027 2028, then the amount of computing reaches levels that start to strain the capabilities of the semiconductor industry. The semiconductor industry still primarily produces CPUs, just the stuff in your laptop, not the stuff in data centers that train AI models, but as the economic value of GPUs rises and rises due to the value of AI models that are going to change, but you know at some point you hit the limits of that or you hit the limits of how fast you can change.

So again I expect there to be a big supply crisis around data centers around chips and around energy and power for both regulatory and physical reasons at some point in the next few years and you know it's a risk, but it is also an opportunity. I think it's an opportunity. to think about how technology can be governed and it is also an opportunity, I will say it again, to think about how democracies can lead. I think it would be very dangerous if the leaders in this technology and the holders of the main resources were In authoritarian countries, the combination of AI and authoritarianism, both internally and on the international stage, scares me a lot.

What about the energy issue? I mean, this takes a huge amount of energy and I mean I've seen different numbers like this floating around very many things they could know in the next few years, like adding Bangladesh to global energy usage or choosing their country correctly. I don't know what exactly all of you are going to use by 2028. Microsoft on its own is opening a new data center globally. every 3 days you have um and this comes from a Financial Times article Federal projections for 20 new gas-fired power plants in the US by 2024 to 2025 there's a lot to talk about this is now a new golden age for natural gas because we have To a large extent, there is a huge need for new energy to manage all this data, to manage all this computing, so I feel like there is a literal question of how to get the energy you need and at what price, but also in a kinder way. of moral conceptual issue of we have real problems with global warming we have real problems with how much energy we are using and here it is, you know, we are taking off on this, you know G really steep curve of how much of If it looks like we need to get into the new AI race , it really comes down to what uses are being put to the model, so I think the worrying case would be something like cryptography, right?

I'm someone who says I don't think you know what energy you know was used to mine the next Bitcoin. I think it was purely additive. I think that didn't exist before and I can't think of anything useful that was created from that, but I don't think that's the case with AI, maybe AI will make solar energy more efficient or maybe it solves controlled nuclear fusion or maybe it makes geoengineering more stable or possible, but you know, I don't think we should depend on in the long term there being some applications where the model does something that used to be automated and that was previously done through computer systems and the models capable of doing it faster with less computing time, those are pure victories and there are some of them, there are others in which the same amount of computing resources are used or maybe more computing resources, but to do something more valuable that saves labor elsewhere.

There are cases where something used to be done by humans or in the physical world and now it's done by models, maybe it does something that before I needed to go to the office to do that and now I don't need to go in there anymore. the office to do that so I don't have to get in my car I don't have to use the gasoline that was used for that the energy accounting for that is a little difficult compared to the food that humans eat or you know, and what is it? the energy cost to produce that, so in all honesty, I don't think we have good answers about what fraction of use points in one direction and what fraction of use points in others in many ways, how different?

This comes from the general dilemma of you know, as the economy grows, it uses more energy, so I guess what I'm saying is that it all matters how you use technology. I mean, my boring short-term answer is that we get carbon offsets for all of this, but let's look beyond that, to the macro question here, but to see the other side, I mean, I think the difference when you say you know that this is always a question we have when we are increasing GDP. It's not entirely a cliché because it's true to say that the main challenge of global warming right now is for countries like China and India to get richer and for us to get richer is a huge human imperative, a moral imperative for the poor in the world. make the world less poor and if that means they use more energy, then we just have to figure out how to make that work and we don't know of a way for that to happen without them using more energy.

Adding AI isn't like it raises a whole different set of questions, but we're already pushing the limits or maybe well beyond the limits of what we can do energetically safely now that we add this and so maybe some of the gains in energy efficiency that we are going to obtain in rich countries. be eliminated by this kind of uncertain reward in the future, you know, maybe through AI we'll discover ways to stabilize nuclear fusion or something, you could imagine ways it could help, but those ways are theoretical and, in the short term, the damage in terms of energy use is real and also, by the way, the damage in terms of Just Energy pricing is just complicated because all of these companies, Microsoft and Amazon, I mean, they all have many of you know goals. of renewable energy now if that is clashing with their Market Incentives it feels like they are running too fast towards market incentives without an answer of how that all works out yeah I mean I think the concerns are real let me back up a little bit, which is again, I don't.

I think the benefits are purely in the future, it kind of goes back to what I said before, like there may now be use cases that are a net energy savings or that, to the extent that they're not a net energy savings. energy, do it through the general mechanism of Oh, there was more demand for this. I don't think anyone has done a good job of measuring, partly because the applications of AI are so new, which of those things dominate or what's going to happen to the economy, but I don't think so. we should assume that the harms are entirely in the present and the benefits are entirely in the future.

I think that's my only point here. I guess you could imagine a world where in some way or another we were incentivizing uses of AI that were Yol of some kind. of social purpose, you know, we were investing a lot more in drug discovery or we cared a lot about things that facilitated remote work or choosing your set of public goods, but what really seems to me to be happening is that we are building more and more powerful models and just throw them into a terms of service structure to say: use them, you know, as long as you're not trying to politically manipulate people or create a bioweapon, just try to figure this out right. , you already know.

Try to create new stories and ask him about your personal life and you know, make a video game with him and you know, if Sora comes out, sooner or later make new videos with him and all that will consume a lot of energy. I'm not saying I have a plan to unite AI with good and in some ways I can imagine that going very, very wrong, but it does mean that for a long time it's like you can imagine the world you're talking about, but that would require some kind of . to plan that thing that no one is involved in and I don't think anyone wants to get involved in it, you know, not everyone has the same conception of the social good, one person may think that the social good is this ideology, another person, you know, We've already seen it with some of the Gemini stuff, but companies can try to create beneficial applications on their own, which is why we're working with cancer institutes.

We hope to partner with African Education Ministries to see if we can use the models in a positive way for education rather than the way they can be used by default, so I think individual companies and people can take measures to direct or tilt this towards the public good. It will never be like that in 100% of what we do, so I think it's a good question what the social incentives are without DCT affecting ideology or defining the public good from above. What are the incentives that could help? With this I don't feel like I have a systemic response either.

I can only think in terms of you know what Anthropic is trying to do, but there's also the issue of training data and the intellectual property that goes into things like Claude like GPT like Gemini, there are a number of copyright lawsuits, yours faces some wide eyes, I suspect everyone faces them now or will face them, and a broad sense that these systems are being trained based on the combined intellectual output of many different people. The way Claude is able to quite effectively imitate my writing is that he has been trained to a certain extent in my writing, so he actually picks up my stylistic tics quite well.

You look great, but you haven't sent me a check about that and This seems like a place where there is a real liability risk for the industry, like what if you actually had to compensate the people you're training on ? And if you and I recognize that I probably can't comment on the lawsuits themselves, but but. I'm sure you've had to think a lot about this and that's why I'm curious how you understand it as a risk but also how you understand it morally, that is, when you talk about the people who invent these systems gain a lot of power and also , a lot of wealth, you know what about all the people whose work allowed them to create images in a million different styles and I mean, did someone come up with those styles like what's the responsibility to get back to the intellectual commons and not just to the commons but to the real wages and economic prospects of the people who made all this possible.

I think everyone agrees that they know that the model should not generate word-for-word copyrighted content for things that are available on the web is publicly available our position and I think there's a strong argument for that is that the training process again, you know, we don't think it's just about sucking up content and spitting it out or you shouldn't spit it out, it's actually a lot more. like the process where you know how a human being learns from experiences so our position is that that's transformative enough and you know I think the law will support this you know this is fair use but those are ways narrow legal thinking.

About the problem, I think we have a broader problem, which is that regardless of how it was trained, it would still be true that we are building more and more general cognitive systems and that those systems will create interruptions, maybe not necessarily one by one. replace humans, but they're really going to change how the economy works and what skills are valued and you know we need a solution to that broad macroeconomic problem. We cannot do as much as I have stated. Legal points I made earlier that we have a broader problem here and we should not ignore that there are a number of solutions.

I mean, I think the simplest one, which I recognize, doesn't address some of the deeper issues here. Do you know things about the type of guaranteed basic income? But I think there's a deeper question here, which is how do you know as AI systems become capable of doing larger and larger portions of cognitive work? How is society organized economically? People find work and meaning and all that and you know Ju? In the same way you know that we went from an agrarian society to an industrial society and you know that the meaning of workchanged and you know it was no longer true that 99 percent of people were peasants who worked on farms and had to find new methods of economic organization.

I suspect that there is some different method of economic organization that will be forced to be the only possible response to shocks to the economy that will be small at first, but will grow over time and we haven't figured out what it is. We need to find something that allows people to find meaning that is human and that maximizes our creativity and potential and flourishes from Ai and as it does with many. Of these questions I do not have the answer to that right, no, I do not have a recipe, but that is what we need to do in some way, but I want to sit between the narrow legal answer and the broad one that we have to do. completely reorganize society's response, although I think that response is really possible over decades and in the middle of that there is a more specific question, I mean, you can even take it from the instrumental side, there is a lot of effort right now to to create search products that use these systems, you know, you can chat.

GPT will use Bing to search for you and that means the person won't go to Bing and click on the website where chat GPD gets their information and provides it. website an ad impression that they can turn into a very small amount of money or they don't go to that website and they don't have a really good experience on that website and maybe they're more likely to subscribe to whoever is behind that website and so on hand that seems like some kind of injustice committed against the people who create the information that these systems are using.

I mean, this is true for perplexity, it's true for a lot of things that I'm starting to see that the eyes are trained to see. or they are using a lot of data that people have generated at a real cost, but not only are they not paying people for it, they are actually intervening in the middle where there would normally be a direct relationship and making this relationship never happens, that I also think in the long run creates a training data problem, even if you just want to look at it instrumentally, if it becomes infeasible to do journalism or do a lot of things that create high quality information, the ability to Okay, the ability to all your businesses to get high-quality, up-to-date, constantly updated information becomes much more complicated, so it seems to me that there is a moral and self-interest dimension to this.

Yes, so I think there can be business models. that works for everyone, not because it's illegitimate to train with open web data in a legal sense, but simply because there may be business models here that offer a better product, so you know things I'm thinking about us. you know that newspapers have archived some of them are not publicly available, but even if they were, it may be a better product, maybe a better experience say talk to this newspaper or talk to that newspaper, it may be a better experience to give the ability to interact with the content and point out places in the content and every time you call that content to have some kind of business relationship with the creators of that content, so there may be business models here that spread the value in the way correct, as you say. about movies using search products, I mean, sure you're checking ads, but there's no reason it couldn't work a different way.

There's no reason why movie users can't pay for search APIs instead of paying for them. advertising and then have that propagate through whatever original mechanism paid the content creators, so that when value is created, money can flow. Let me try to finish by asking a little bit about living on the slope of the curve that you think we're on, do you have kids? um, I'm married, I don't have kids, so I have two kids, I have a 2-year-old and a 5-year-old, and particularly when I'm doing AI reporting. I really sit in bed at night and think about what I should do here with them, what world I'm trying to prepare them for, and what is needed in that world that is different from what is needed in this world, even if I think there is some possibility. true and I think there is some possibility that everything you say is true.

It implies a very, very, very different life for them. I know people in your company with children. I know you're thinking about this. How do you think that? I mean, what do you think should be different in the life of a 2 year old? Who is experiencing the pace of change that you are telling me is true? If you had a child, how would this change your way of thinking? The very short answer is: I don't know and I have no idea, but yes. Trying anyway is the right thing to do, people have to raise their children and they have to do it the best they can.

An obvious recommendation is to simply be familiar with the technology and how it works, like the basic paradigm of "I'm talking to systems and systems." they're taking action on my behalf, obviously, as much familiarity with that as possible. I think I think it's useful in terms of: Do you know what kids should learn in school? What are the careers of tomorrow? I really don't know, do I? Take this to say, well, it's important to learn programming and AI and all that, but you know AI will have an impact on that too. I don't think he likes any of that, possibly first, yeah, sure, possibly first.

He seems better at coding than other things. I don't think it works for any of these systems to do one by one what humans are going to do. I don't really believe that way, but I believe. It may fundamentally change industries and professions one by one in ways that are hard to predict, so I feel like I only have clichés here like getting familiar with technology, teaching your kids to be adaptable, and being prepared for a very changing world. quickly I wish I had better answers, but I think this is the best I got. I agree, it is a very good answer.

Let me ask you the same question a little bit from another direction because one thing you just said is get familiar with the technology. and the more time I spend with technology, the more I fear that what I see when people use AI around me is that the obvious thing that technology does for you is automate the first parts of the creative process, the part where you are supposed to If you're reading something difficult, I can summarize it for you, the part where you're supposed to sit there with a blank page and write something well, the AI ​​can give you a first draft and then you have to check it and make sure. you know it actually did what you wanted it to do and check it out but I think a lot of what makes humans good at thinking comes in those parts and I'm older and I have self discipline and maybe this is just I like to hold on to an old way to do this well.

You could say why use a calculator from this perspective, but my real concern is that I'm not sure if what they should do is use AI a lot or use it a little. For me it is actually a very big road, right? Do I want my kids to learn how to use AI or be in a context where they use it a lot or do I really want to protect them from it as much as I can? so they further develop the ability to read a book quietly on their own or write a first draft. Actually, I do not know.

I'm curious if you have an opinion on this. I think this is part of what makes the interaction between Ai and a complicated society where sometimes it's hard to distinguish when an AI is doing something that saves you drudgery versus doing the interesting part I I I I will say that time and time again you will get something technological, some technological system that does what you thought. It was the core of what you're doing and yet what you're doing turns out to have more pieces than you think and they kind of add up to more things, it's like you knew before, I used to have to ask for directions.

I got Google Maps to do that and you know you might worry. Do I trust Google Maps too? Do I forget the environment around me? Well, it turns out that in some ways I still need to know, to have an idea of ​​the city and the environment around me. It's like it reallocates space in my brain directly to some other aspect of the task and I suspect I don't know internally within the anthropic as one of the things I do that helps me run the company. is, you know, I'll write these documents about your strategy or just some that you know, thinking in some direction that others haven't thought about and you know, of course, sometimes I use the internal models for that and I think what.

I found that yes, sometimes they are kind of good for conceptualizing the idea, but in the actual Idea Genesis I just found a workflow where I don't use them, so they are not that useful. for that, but they're useful for figuring out how to express a certain thing or how to refine my ideas, so maybe I'm just saying I don't know, you just find a workflow where the thing complements you and if not, it happens naturally. , in some ways it still happens again if the systems become general enough, if they become powerful enough, we may have to think along other lines, but in the short term, at least I've always found that maybe that's too optimistic , maybe I'm too optimistic.

I think that's a good place to end this conversation then, although obviously the exponential curve continues, so always our final question: what are the three books that you would recommend to the audience? So yes, I have prepared three, although they are all current. in some cases indirectly, the first one will be obvious, it is a very long book, the physical book is very thick, but the making of the atomic bomb, Richard Rhodes, is an example of how technology develops very quickly and with very broad implications , just looking through them all. the characters and how they reacted to this and how the people who were basically scientists gradually realized the incredible implications of the technology and how it would lead them to a very different world than what they were used to.

My second recommendation is that it is a science. fiction series The expansion book series, so initially I watched the show and then I read all the books and you know, the world it creates is very advanced in some cases, it has a longer lifespan and the humans have expanded to space, but You know, we still face some of the same geopolitical issues and some of the same inequalities and exploitations that exist in our world are still present, in some cases worse, that's the backdrop and you know the core is about something fundamentally new. technological object that is being brought into that world and how everyone reacts to it, how governments react, how individual people react, how political ideologies react, and so, you know, I don't know when I read that a few years ago.

I saw a lot of parallels a while ago and then my third recommendation would actually be The Guns of August, which is basically the story of how World War I started, the basic idea that crises happen very quickly, almost no one knows what is happening. passing there. a lot of miscalculations because they are humans at the center of this and somehow we have to learn to step back and make wiser decisions in these key moments. He said that Kennedy read the book before the Cuban Missile Crisis and I hope that our current policymakers are at least thinking in the same terms, um, because I think it's possible that similar crises could be coming our way.

Dario Amade, thank you very much, thank you for inviting me. This episode of the Ezra Kin Show was produced by Roland, who reviewed by Michelle Harris a senior engineer is Jeff Geld our senior editor is Claire Gordon the show's production team also includes Annie Galvin Kristen Lynn and Aman Sahota original music by Isaac Jones audience strategy by Christine Sami and Shannon Busta, executive producer of the New York Times, thinks the audio is Anaser and special thanks to Sonia Herrero.