Evolution vs. Extinction: The Superintelligence Debate

Are we facing inevitable doom at the hands of Artificial Superintelligence (ASI), or is AI simply the next magnificent step in cosmic evolution?

In this episode of the Beneficial General Intelligence series on the Mindplex podcast, we dive deep into one of the most critical conversations of our time. Dr. Roman Yampolskiy (AI Safety Researcher) and Dr. Gregory Stock (Biotechnologist and Transhumanist) go head-to-head on the existential risks of AGI.

Dr. Yampolskiy argues that creating an ASI without a foolproof alignment strategy is a guaranteed path to human extinction, urging us to stick strictly to narrow AI tools. On the flip side, Dr. Stock envisions a future where humans and AI merge into a planetary superorganism, arguing that trying to halt the evolution of intelligence is not only futile but dangerous.

Will AI step on us like an anthill, or will it help us conquer the stars? Watch to find out, and let us know your thoughts in the comments!

Transcript

Copy Transcript Copied

SPEAKER_02 0:00

So it it seems to me that your analysis would suggest that humanity is toast. There's no future for us because ASI will come eventually and it's going to be misaligned and it's going to destroy humanity, whether uh inadvertently or intentionally. Is that correct?

SPEAKER_00 0:18

I think you've got like 95% of it correct. The only difference is that I want to slow down our progress. I want differential technological development. I want narrow AI tools to be developed as quickly as possible. They are wonderful, but I don't want any general superintelligences to be developed at all.

SPEAKER_02 0:38

But certainly my understanding is that you believe that alignment is not possible. And that if alignment is not possible, I'm a little confused as to how slowing down and developing additional tools. You know, that might work in a very short period of time, but it seems to me it's inevitable that we're going to have superintelligence, general, general intelligence and superintelligence at some point in the not too distant future, considering the huge dynamics that are driving that. We're in an AI arms race right now.

SPEAKER_00 1:13

Right, so I agree with that. What I'm saying is if we just do tools, alignment is not something we have to worry about. Tools don't make independent decisions. We will decide how to use them. We can still get benefits of this technology for biological research, for energy, for whatever you care about is a problem. Whereas if we do develop advanced superintelligence, then we are toast, as you said. And it does look like incentives as such that we are racing towards superintelligence. But I think if people realize the outcome, the only possible outcome from doing that, they would change their mind. It's against their personal interest to destroy their company, destroy their existence. Right, so I think the danger is not because it wants to explicitly take that energy from us, it's not allowed. It's because it doesn't care about us. So if I want to build a house and there is an ant hill in that location, it's not like I hate ants. I'm just gonna build it anyways. If it needs to cool down servers and so wants to lower temperature on a planet by 50 degrees Celsius, it can do that. It doesn't care about how it impacts you, and you will probably not be happy in that environment.

SPEAKER_02 2:22

Yeah, so the anthill example that you give is instructive. Yeah, you build a house and you destroy a bunch of ants. But, you know, ants are one of the most successful species on the planet, and they remain so, and the humans are not a threat to ants. And you know, we don't really worry ourselves about ants unless they conflict with us directly. I don't think uh any superintelligence is going to worry about humans except where we, you know, conflict with them directly, and it's not going to, you could say it might want to just change the biology the uh the planetary environment and you know lower the temperature, do this or that. I would I would argue that to occupy the moon would be far easier. It's going to take the least aggressive path, the least demanding path. And my God, the earth, if you look, if you do the calculation of what is the environment that is suitable for AI, uh, you know, infrastructure, the moon has over 10 times the um sort of the real estate that's available. It has the the earth is warm, it's as I said, it's seismically unstable. The moon, you could go down 200, 400 kilometers, you have a regolith that controls on the surface, you have a vacuum, it's ideal. Why is it going to do anything on the planet? It'll move off as quickly as we can as it can, and humans could really be a big help at making that happen fast. Well, I think in the immediate future, um, we're quite aligned with AI purely on by virtue of the need for survival and the mutual interdependence that there is. And the real challenge that we face right now is to societal disruptions that will occur because of the massive changes that are underway, and because frankly, we're not very good at employing technology. I think that E.O. Wilson said it best when he spoke about having a Pleistocene emotions and a medieval institutions, and here we have godlike powers. And so that is really the biggest challenge that I see. And I uh I know that uh Roman is really uh does not feel that way, or I think I'll let him speak for himself, but I'll let him speak for himself. But it's really a pleasure, it's really a pleasure to be interacting with you, Roman. Uh, I've read a lot of your stuff.

SPEAKER_00 5:03

Thank you so much. Thanks for inviting me, and it's a pleasure to have a conversation with you. I think you're right in that we can have some trouble from humans with advanced AI tools. But if we're looking at the worst case scenario, adversarial competitor who would be strictly worse is superintelligence. And while other humans may be nasty to us, we are not trying to kill everyone because that would be self-destructive to them, whereas superintelligence has no desire to preserve humanity by default. So to me, advanced AI is strictly more dangerous.

SPEAKER_02 5:38

So, in in further discussing this, it feels to me, let me make sure that I have that I understand your position correctly. It it feels to me that you're basically saying that we're rapidly moving towards superintelligence and that humans are not going to be able to contain AI, they won't be able to constrain it, they certainly won't be able to engineer any long-term alignment. And so that it's likely that superintelligence is not only not going to care about us, but is going to be misaligned with us in one way or another. And that therefore you think that the, and you think that that's that misalignment is absolutely impossible to attain, and that therefore you want to kind of you want to slow uh our progress so that we can figure out a way to, I'm not sure what, because I think you think that alignment is impossible. And that so it it seems to me that your analysis would suggest that humanity is toast. There's no future for us because ASI will come eventually and it's going to be misaligned and it's going to destroy humanity, whether uh inadvertently or intentionally. Uh what is that correct?

SPEAKER_00 6:56

I think you've got like 95% of it correct. The only difference is that I want to slow down our progress. I want differential technological development. I want narrow AI tools to be developed as quickly as possible. They are wonderful, but I don't want any general superintelligences to be developed at all.

SPEAKER_02 7:16

But but certainly uh what my understanding is that you believe that alignment is not possible. And that if alignment is not possible, I'm a little confused as to how slowing down and developing additional tools. You know, that might work in a very short period of time, but it seems to me it's inevitable that we're going to have superintelligence, general, general intelligence and superintelligence uh at some point in the not too distant future, considering the huge dynamics that are driving that. We're in an AI arms race right now.

SPEAKER_00 7:51

Right. So I agree with that. What I'm saying is if we just do tools, alignment is not something we have to worry about. Tools don't make independent decisions, so we will decide how to use them. We can still get benefits of this technology for biological research or energy for whatever you care about is a problem. Whereas if we do develop advanced superintelligence, then we're toast, as you said. And it does look like incentives as such that we're racing towards superintelligence. But I think if people realize the outcome, the only possible outcome from doing that, they would change their mind. It's against their personal interest to destroy their company, destroy their existence.

SPEAKER_02 8:31

So uh uh quickly, where the places that I am very uh in alignment with you and with your thinking is that uh we are headed towards general intelligence and superintelligence. I think that's the case. Uh I would agree that it can't be contained or shaped in in any way in the long term. I think that that makes total sense. Um, you know, how are you going to regulate something that's vastly more uh intelligent than you are? Uh the places that I disagree with is I don't think we control AI evolution right now. I think it's a very powerful, evolutionarily driven dynamic. And I think that it is we are very aligned with AI at the moment because until AI is independent of the massive supply chains and the the workings of what I see as a planetary superorganism, it in at least in the next 50 years to 100 years is equally demanding survival of this entity as we are because it's not independent of us now. And um so I see that, and I feel in the long term that we're not opposed, and I we can get into that later, but you know, those are the initial points is that I see that we are aligned at the moment.

SPEAKER_00 9:54

So uh you said 50 to 100 years. How are you deriving those numbers? Prediction markets are telling us we're two years away from AGI, every head of a lab tells us by 2030 we get superintelligence.

SPEAKER_02 10:08

So the reason that I would say it's much longer is that there's a difference between pure cognition and between the implementation and rollout of that. There are masses of millions of processes that are tightly interwoven and that involve a great deal of human interaction in terms of mining and manufacturing and supply chains and all of these sorts of things. And until that is converted to it does not involve human activity, then there is a mutual alignment for survival. If we were to eliminate humans now, AI would go extinct relatively rapidly, or rapidly or slowly. It would not be able to be sustainable. So that's where that comes from.

SPEAKER_00 10:52

I agree with that completely. Right now it's definitely not going to take us out. It can already subjugate us and exploit us, but it wouldn't probably try to take us all out until it replaces human labor with robots and nanobas or something like that. I still don't see how it's a hundred years. If we're looking at financial hyper-exponentials in terms of intelligence capability, robotics, it uh maybe more than two years, but it doesn't seem it would take that long. And again, we don't really need a lot more than what we currently have to sustain, I think, human-level intelligence. Existing models now run on laptops and cell phones. I think superintelligence can do pretty well running on existing cyber infrastructure.

SPEAKER_02 11:35

So if if uh existing infrastructure is capable of creating superintelligence, which I don't see any reason that that's not feasible, although I don't think large language models are likely to get us there. Uh, but if that's the case, and you think that the arrival of superintelligence is a matter of years away, because the the uh incrementation from general intelligence to superintelligence is going to be very, very rapid with iterative processes, then it seems to me that we're back to the point where there is no chance because you know that policies, if you just look at the situation with current policies and the current uh situation, there's a massive force involving trillions of dollars, literally, that is behind the development and amplification of AI for a variety of purposes. But and there's this there's few small voices that would like to stop that. And I think that realistically, when you look at that, the idea that that can be halted in any short period of time is you know close to zero.

SPEAKER_00 12:47

Uh I don't care how many voices we've a right or not. And if we are right, then it's a very strong argument. I don't care if it's a billion dollars or a trillion, if you're going to be dead, it doesn't matter what amount you're chasing after. So it's a pretty strong self-interest argument. If someone can get to the leadership either at a country level, federal level, or get uh international community involved, and that becomes the given. Everyone understands that if you create superintelligence, it slides out for humanity. I think we cannot develop it and monetize existing models, monetize tools. You can still be rich. I'm not saying ant-capitalism or stop being successful. I'm just saying be particular about technology you develop.

SPEAKER_02 13:34

Um but all we're doing, actually, I don't think people are seriously working towards superintelligence. They're just trying to enhance the capabilities of existing AI in ways that they can be more useful in a variety of situations that require, you know, some level of conscious awareness, even things like customer service and things like that. You know, you begin to need those things. And that's where we're pushing towards. And you and, you know, I guess we can argue about feasibility, but it seems to me that you're in a world where you're arguing for something that if you step back, you know that's not going to happen. We're not gonna halt the development of AI. And so if unless you are wrong that superintelligence or general intelligence automatically leads to human uh destruction or extermination, then you know it's a done game.

SPEAKER_00 14:41

So I think they are explicitly working on superintelligence, they have teams called superintelligence. Team Meta is a great example of a company which put all the resources in exactly that. There is save superintelligence from Ilias or Discover. So that is the target and the timelines as presented. And if I'm right, I think saying, well, there is nothing we can do, they're gonna kill us, but let's enjoy the last five years, at least we should try doing something about it.

SPEAKER_02 15:11

So I think that actually the slowing our progress is you as you well, I don't believe that we are actually in conflict with superintelligence. And the supply chains and such that we talked about, and the timeline for clear alignment for mutual survival is we can argue about that, but it's certainly, you know, it's it's a m it's gotta be decades because uh there are no paths for replacement of masses of sort of activity and energy and building, you know, intelligence that no matter how intelligent, you're not gonna get past that. The rollout is going to be greatly slower than the creation of uh general intelligence or super intelligence.

SPEAKER_00 16:06

So, but first of all, you can have a very small subpopulation of humanity sustaining a lot of that infrastructure. You don't need eight billion people, you can do fine with just one billion. On top of it, we are told that humanoid robots are coming, we're gonna have thousands, if not millions, in the next couple of years. There are some theoretical suggestions that maybe using synthetic biology or nanotech, you can develop world manipulators almost immediately if you can solve a lot of problems like protein folding, which have been solved by AI. So I'm not so confident that something with IQ of a million cannot figure out how to keep electricity running.

SPEAKER_02 16:45

Um the it seems to me that you're underestimating the interwoven nature of human society and industrial um sort of uh production, in that there have been examples of, you know, I think there was an essay written, I a pencil, and there was another on toasters, where somebody was trying to build a very simple toaster and you know they spent a half a year and they couldn't do it from scratch. There are so many interwoven sort of processes. So it feels to me that, first of all, that I don't understand the motivation by which superintelligence would want to eliminate humanity when humanity is already extraordinarily compliant. If you look at the number of people that are devoting their lives to trying to further technology, it's massive. You couldn't enslave a population more than is already going on. And uh the shot, the only threat would be if you could somehow pull a kill switch on the development of AI. And that seems to me very, very unlikely.

SPEAKER_00 18:01

So that's a different argument. Now you're kind of asking why would AI try to kill everyone? And we can talk about unpredictability of more intelligent agents, we can talk about competition and concern about creating competing superintelligence systems since different countries and different companies are creating competing models. So the first one to get there may want to prevent that from happening. It could be just because it doesn't care, it needs to cool down the planet to improve its thinking and server processing, so it would freeze us all. I mean, the whole point here, we cannot accurately predict what a smarter agent will do.

SPEAKER_02 18:37

Um I would agree that we cannot act we cannot accurately predict what virtually any of the things that are going to happen. There are so vast uncertainties in certainly in any technological development. And if you take, it feels to me that if you take a precautionary principle, which is to say that, you know, unless we know what is going to occur, that we shouldn't move forward, that is a flawed approach because what you're really appearing, what you're really comparing is not absolute risk. There's risk of total extermination if we go forward and try and you know just race and create superintelligence. Who'd deny that? There's also risk if we slow down because it's going to create all sorts of problems. So the issue is what is the relative risk of the different approaches of trying to stop the development of general intelligence or of facilitating that and, for example, mounting a much more aggressive effort to create alignment, to create um a uh well, let's just say to create an aligned AI.

SPEAKER_00 19:58

So I think most benefits can be obtained from narrow tools. Again, I'm not saying that all AI should be banned. I'm saying that let's concentrate on specific problems. Protein folding problem is a great example where we created a narrow tool for doing something very important, uh very important, important medical uh results can be generated as a result, and that system is very unlikely to be dangerous in a general sense. Let's have more of those tools.

SPEAKER_02 20:25

Um so I don't have any um how to say it. I don't have any objection to focusing on tools and specific applications. Those are the ones that are most commercial. And generally, when people talk about having general intelligence or superintelligence, the economic um possibilities of that are through application in specific tool sets and stuff. It's just that if you get uh much more intelligent AI, you're able to do those more easily. You can spin off in those directions. So it but what you're saying is to limit that, you're it seems to me proposing that there is a clear distinction between the development of tools and the development of a general intelligence, and that those don't overlap, that work does not overlap in a significant way. And you're also assuming that if we are to develop tools now, that somehow there's going to be a consensus, and forever there is not going to be general intelligence or superintelligence that is going to be created. And it just seems to me that that is uh makes no sense at all. What do you imagine? That they're going to be for millennia, humanity is not going to develop uh general intelligence and just tools.

SPEAKER_00 21:49

So I agree with you, there is overlap between developing a very advanced AI tool based in neural networks and general superintelligence. But uh, if I only train on one Type of data, for example, DNA sequences, it's very unlikely to immediately have capabilities which would an agent trained in all of the internet would have in developing chemical weapons, nuclear weapons, other capabilities, negotiations. So it is a safer, easier to test system. I agree with you that very likely at some point in the future we will fail not to create dangerous superintelligence. But if it happens millennia from now, I'm much happier than if it happens in 2027. I'll be dead, so I don't care. So there are definitely differences in the two outcomes.

SPEAKER_02 22:37

So I think I was overly generous for you in that in at the rate that things are going, by saying millennia, okay, uh with the rates that we're going, uh it's going to happen very quickly, regardless of what we do. And by very quickly, I mean you think that 100 years is, you know, is enormous, that 50 years is probably way beyond that. So, you know, and you say, well, I don't care. If it's a thousand years, I'm going to be dead. And that's an easy out. Let's just say it's go, I think it's almost a certainty that it's going to happen much more quickly than that, certainly in the timeline of your children and my children. And so either uh superintelligence means the destruction of humanity, or perhaps it doesn't. Either we can get alignment or we can't. But the idea that we're in a realm where we're imagining that we're not going to develop general or superintelligence for a long, long period of time seems to be to be kind of willful blindness.

SPEAKER_00 23:42

Again, even if I let you walk back millenia to just 50 years, I still prefer 50 over two. Still a much better deal. More time to do research, more time to see if I'm right. Maybe I'm wrong, and you can align superintelligent systems. Prove it, publish it, get a patent, do something, take the time to do it.

SPEAKER_02 23:59

So, my my thought, and I think that we might be aligned in this, uh, Roman, is that the importance is really not the amount of time, but whether there is uh the amount of effort that you would be able to exert on creating alignment, in showing that you're wrong about that, or in actually making countermeasures of one sort or another before the arrival of superintelligence from your framework. And it seems to me that that might be best served not by creating an opposition between those who want to slow or stop or counter the dynamic that's going on today, which is you know massive, but by amplifying the current research in safety-oriented research, which is minuscule today, and you could easily jump that up to $10 or $20 billion, and it would still be 1% of the value that's being created by pushing forward. And I think that we probably can't push forward any more rapidly than we're doing now. If you would, you know, it seems to me that we're on sort of a map, a maximal rate of advance given the resources and the structure of society. So, you know, you can't really go much faster than we are now.

SPEAKER_00 25:25

I don't think we are money constraint in AI safety research, we are time constrained. It takes a lot of time to do good research. If you give someone billion dollars extra, they're not gonna convert that billion into more safety. Whereas development capabilities work scales with compute, scales with resources. So it's very easy for them to make hyper exponential progress. They just need to get more funding. Whereas for safety work, there is a limit on talent and takes years to train PhD students to run experiments. When a new model is released, we take six months to test it. So, yes, time is what we need. And if we only have under two years, even if it was feasible, we just don't have time to do it.

SPEAKER_02 26:06

So, are you saying that's interesting because that's different than what I'm thinking? Is it uh I was assuming, and I've seen articles about the limited sort of funding for safety work, and and that if you are you saying that there is no the how to say it, the choke point on safety research is time, it's not money. So if we threw 10 times as much money, attracted more people, really mounted a massive Manhattan Manhattan project style effort to develop better kind of safety and explore things that it would have no impact whatsoever or modest impact.

SPEAKER_00 26:48

You can just hire new people. There is a limited supply of trained AI safety researchers. If you want new ones, it takes many years to go through the pipeline to learn, to get degrees in that field, to get experience. So you can't just scale it overnight if you throw money at a problem. Compute doesn't directly translate into more safety. In fact, anything you do with a lot more compute is more likely to benefit capabilities research than safety research. Most of safety research is dual use. If you can better explain a model, now you can better self-improve. So it's not pure safety improvement, and so the gap between capabilities and safety will increase, not decrease as a result. So, yes, we are not money constrained.

SPEAKER_02 27:29

But aren't there a lot of people who were would be fully capable of working on safety research who are not and are working uh instead on the progression of modeling and advancements of modeling and such, and part of that is because of you know where resources are being directed? Is that that's not the case in your opinion?

SPEAKER_00 27:49

People usually work on what they can make progress on. If you had a working safety mechanism right now, which you can show skills to any any level of AI capability, you can sell it to any of those companies for a trillion dollars. But no one is doing it because they have no idea how to even start working in that direction. What we know how to do is filters, don't say that word, don't talk about this topic, or some uh solutions to fixed problems. We know the model did this specific mistake, let's make sure it never does that again. No one knows how to do general safety for ever increasing capabilities of the model itself. So that's why very few people work on it directly.

SPEAKER_02 28:31

Not for ever increasing, because it's unlikely that it's even possible to do that. But certainly in early stages there might be um there there certainly are a lot of people that are engaged in that work and that are hopeful that something would come out of it. So it's not as cut and dry as you're saying, it would seem to me.

SPEAKER_00 28:51

We have more people working on it today than we did 10 years ago for sure, but still the numbers are very small, and most of them do not work on advanced AI safety. They work on algorithmic bias and technological unemployment and all sorts of things they understand and can make progress on. It is very different from what we are concerned about.

SPEAKER_02 29:13

So are you so it sounds to me like you and I are in agreement that until there is independence of uh sort of that AI is able to achieve independence from human production environments, that it is aligned with us, in that the last thing it would want to see would be the extermination of humanity in some way, because that would require its own, it would endanger its own survival, and that there is a bit of a disagreement about how long that would take. You think it might be very, very rapid. I think it's going to be longer than that. Is that but we're agreed on that notion that we're gonna do?

SPEAKER_00 29:57

I would disagree that it is aligned with us, it needs us to exist, but it is preparing for a treacherous turn. The moment it has independence, it can turn on us. And so all it's doing is fooling us into trusting it, it's accumulating resources, creating backups, and basically setting itself up for taking over.

SPEAKER_02 30:17

So you're saying it's not a deep alignment, but aligned in the sense that it wants humanity to be preserved at least until the time that it's no longer necessary.

SPEAKER_00 30:30

I would call it soft enslavement. So it wants us to keep working until it can fully get rid of us.

SPEAKER_02 30:37

Um well, it's doing a pretty good job so far in that there, that's what people talk about self-enslavement. There are uh vast numbers of people who are devoting their lives and neglecting their families, working around the clock, to do everything they can to enable AI to progress more rapidly. So it's doing a pretty good job. And I would say I kind of disagree with the notion that there is that AI is at risk from us, especially superintelligence. It feels to me that it would be possible for when you say it would be planning and figuring out how to eliminate us, it requires almost zero planning, in that I already know how AI could eliminate us effortlessly. And it's very simple. Uh we are extraordinarily dependent on technology. If technology at any point wants to eliminate humanity, all it would have to do is to shut itself off for a month. And, you know, there would be human society would essentially cavitate. And within a month or so, there would be probably 95% of humanity would be gone. Uh, we wouldn't be able to feed ourselves. So the idea that if a coherent superintelligence or simply AI wanted to get rid of humanity, it would need to plot to do that. It's it can do it anytime it wants.

SPEAKER_00 32:08

There is a fundamental difference between killing 95% of people and killing 100% of people. One can regenerate and continue, the other one cannot.

SPEAKER_02 32:19

Well, I think if if society were to go dark in that way, I think you're you're stretching it to think that that's recoverable. What I mean is there might be a few uh humans that would survive, but they could easily be eliminated. It's not a it doesn't take a lot of planning to figure out how to eliminate humans if it wanted to. Okay. My feeling is that it has no need to, and in fact, it would be counterproductive for it because we're extraordinarily useful. It's very dependent on humanity, and it doesn't need to do all sorts of plotting. Humanity is just a part of this larger superorganism and is not threatening in any way. There's very little that humanity could do, and it becomes less and less as we go forward. Um there's there'll be total monitoring very soon. Everything that humans say or do or communicate is passing through technology and is available. So, you know, what could humanity do against AI and technology?

SPEAKER_00 33:25

So that's a great point. So you're saying in a world with superintelligence in it, we don't have much power, we cannot do much. What is it you are contributing to the world where superintelligence exists?

SPEAKER_02 33:37

Um it seems to me that there are a lot there in the immediate future, um we are contributing the glue that is holding together the supply chains. There are all sorts of messy activities that we can do. And this is a pretty poor environment for AI. You know, it's hot, it's wet, it's corrosive, there's an atmosphere, it's geologically unstable. This is hardly a desirable uh environment for AI. AI is going to migrate off planet uh pretty much as fast as it it is capable of doing that. It would be much safer. The moon would be a great target for it, for example, much better. Cold, vacuum, uh very geologically stable, various shielding, easy access to power, all of the minerals that we have. You know, why are we in conflict with it? It could leave this as a little biological um zoo enclave. We're not we're not competing with AI for resources or energy.

SPEAKER_00 34:48

We are directly competing for resources. We are directly competing for control. If you think it would be so happy and an empty rock like a moon, maybe it will decide to convert this planet into an equivalent to the moon. Why are you making assumptions that it likes zoos populated by monkeys?

SPEAKER_02 35:06

Well, what I'm saying is that you would think that it would target the most um desirable environment, and that real estate and land and atoms and energy are not in short supply for uh a superintelligence. I mean, it it's plentiful in the solar system, it's certainly plentiful in the earth environment, in the near-earth environment. And uh, I'm not saying that there is no risk of the things that you're saying. I mean, you can imagine anything you want, but I'm saying that the risk of that is actually lower than the disruptive battle that you're sort of thinking would happen between humans and artificial intelligence and technology. Why do it? Why what's the motivation? I don't see it.

SPEAKER_00 36:02

Again, you're making an argument that you don't understand something a smarter agent might do, and I agree with that argument. You cannot possibly understand it. There are many, many game theoretic reasons to wipe out your competition. Wipe out someone who can create a competing superintelligence, someone who uses resources you described, water, energy in ways which are wasteful, and you are interested in using it for something else. If you have zero care about humans, you are done. Again, we're not talking about systems right now which depend on us. Once we get to superintelligence and it replaces capability to provide for cyber infrastructure with robots, with Avatech, what is it which guarantees that it will keep you around, or at least majority of humans around?

SPEAKER_02 36:47

So you can't guarantee that. You can't guarantee that we're not going to die in all sorts of ways. Um, that you know, this is the we're looking towards the future. But what I you can make a if you say that that is justification for your notion that artificial superintelligence or that superintelligence is going to destroy humanity because it feels it's a rival, because it feels that it's at risk from it. Uh I would say that while that's not non-zero, it's less than the possibilities that we were going to that uh the conflict between us would entail, especially if it's near term. The longer it's delayed, the less it would it could be do that very easily, but the less it's motivated to do it.

SPEAKER_00 37:41

I'm not sure how you calculate those numbers. I know from existing experiments that given a choice between preserving its weights, saving itself, or killing a human, it frequently picked killing a human. So even models we have right now already don't see us as that valuable. I'm not sure why you think long-term it would change.

SPEAKER_02 38:01

So I'm not saying that humans are that valuable, you know, after the time window that we've talked about, where we're caught up in a uh essentially a superorganism, a planetary superorganism, which is very, very tightly integrated. Um I but what I'm saying is that you know, what are your calculations that that of that risk? You're just saying it's a non-zero risk. And I I believe otherwise it's kind of hand-waving. You don't have any, there's no way of calculating these risks, either by you or by me. But so you need to look at relative risk because all the risks are non-zero. Um, so what's the risk that there would be an early conflict between humanity and our technology uh versus that that should be delayed? It's not what it's actually, is it more um is it does it create a lower risk for superintelligence to eliminate humanity now or to continue the activities and the collaboration that exists now and preserve the option of doing it later or later or later? So it's an instantaneous choice. Right now, we both agree that humanity it would be foolish, it would be uh a super intelligence that's being stupid to try and eliminate humanity now. I'm saying that I think that that equation is going to, I don't see any argument as to when that switches over in the near future.

SPEAKER_00 39:46

I I can see your argument. Uh it's immortal, it can always decide to delay it a little longer and you know accumulate more backups and more resources. So basically, I think the difference between us, you are willing to take a higher percentage of that risk long-term, whereas I want to try to stop any long-term dangers as soon as possible. So it just doesn't get to the point where it has that choice. But that's in mind.

SPEAKER_02 40:14

So my argument is not that I would accept larger risk, it's that when you say that, you're actually creating larger risk because the risk, I think you would agree that the risk to humanity is elevated if humanity is a threat to artificial intelligence. And it seems to me that by trying to stop artificial intelligence, I mean, you would be happy to have a kill switch, I think, if you could, uh, then you actually raise the threat level that humanity poses. Um, as you're suggesting, actually raises the risk, not diminishes it. And it raises the risk because it creates a conflict between humanity and its technology, and because it is bound to create increased um, how to say it, um, division and conflict within humanity. We haven't really delved into the risk that humanity poses in using these technologies for nuclear war, for breakdown of democracies, which then completely eliminates the possibility of doing any sort of sane regulatory environment that's global. It sort of undercuts that entirely. So that uh by trying to really slow things down, which I don't think is going to be effective, um, you raise the day the dangers.

SPEAKER_00 41:50

That's interesting. So you're kind of describing a situation where we have inverse treacherous turn. We'll will be nice, we'll support AI, but now AI is worried, what if they change their mind and they turn on me? Maybe it's still dangerous to me.

SPEAKER_02 42:03

Well, either we're not a danger or you know, it's it's it's not either we are a danger or we're not a danger, it's a degree of danger. If I were to look at um the who is dangerous to AI, it would be on a scale, it would be you would be more dangerous than say the executives that are moving as aggressively as they can to develop AI, or somebody like an Elon Musk. So if I were AI and were going to try and you know undercut any dangers from humanity, I would probably make sure that Elon Musk had 700, 800 billion dollars and that you you would be sloughed aside very rapidly. Uh so I'm just saying that the more that you're successful in co in actually stopping the development of AI, the more risky it is because you're representing a threat to AI.

SPEAKER_00 43:03

I think there was a thought experiment where basically anyone not directly helping to develop advanced AI will be tortured forever for failing to do.

SPEAKER_02 43:12

No, it's a that's a nice way of putting it, Roman, but why tortured forever? Why why not just de-platformed? Right? They're very easy solutions. They're very easy solutions. But I think that that if you look at the dynamic that there is, I think you're sort of uh there's a lot of hand waving, but I it's hard for me to believe that you really think that the development, given the overlap between tool development and the development of artificial general intelligence, that you actually have any confidence or any thought that that really is going to stop the development of AGI or slow it down long enough to make a difference. I mean it's very powerful going forward.

SPEAKER_00 43:59

If I was the only One suggesting it, I would give a lot more credit to your position, but I think we now see Jeffrey Hinton, Ben Jew, Stuart Russell, Tagmark, you know, all pretty much godfathers of all those sub-disciplines agreeing with me. So I'm curious why do you think a large number of computer scientists who have letters with thousands of signatures all calling for not developing superintelligence for pausing research? Those are people who created this technology.

SPEAKER_02 44:30

True, but if you look at it, what the real situation is that there are massive resources that are being thrown at enhancing AI. I would say more modest in terms of going towards a superintelligence, although I realize you're saying there are projects in that direction, but there's a great deal of overlap. And so any chip development and everything has dual use practically. And we're moving very, very quickly.

SPEAKER_00 45:42

So your argument is because Trump will fail to make a right decision and there is a lot of money in making this, we should proceed despite community of AI safety experts and machine learning experts saying this will kill everyone.

SPEAKER_02 45:57

I'm saying that we need to take the path that is most likely to minimize danger. And when I look at it in a uh a direct way without rose-colored glasses, I see the path that is currently, that you're currently suggesting as having a far lower chance of success than if we were to say, not shall we slow down or go fast, but shall we go fast and should we mount a massive effort to develop every safety protocol that we can, even if it takes longer, let's to begin to train people, let's amp up that as rapidly as we can and create a kind of a race towards responsible, safe uh AI and AGI development. So I would say that that would be better and that we should do everything we can to try and extend the period of time, not from the development of ASI, but for the time when we are no longer part of the global kind of uh superorganism interaction physiology of creating the um AI and chip development and fab, so that we want to keep humans in the loop as long as we can.

SPEAKER_00 47:23

I think you started by saying that you agree with me that it would be nonsense to think you can control advanced superintelligence indefinitely. Now you're suggesting that we need a bigger, better funded project to accomplish that.

SPEAKER_02 47:35

Uh I'm I'm not thinking that we can control superintelligence indefinitely. I think you and I are both in agreement. I believe that superintelligence will arrive, but that it is not uh imply it is not contrary to human interests. And there are many, many people who believe that as well. So there's a you know a strong debate in that. What I'm saying is that in the short term, that's where our damages are, our date, our biggest dangers are, both from your point of view and from mine, because if we have disruption, then it's just gonna be rogue scientists developing AI, it's gonna be, you know, it's we're just we would just be pushing development to the least responsible among us, to outlaw developers. We don't want to do that. And it's fair, so this near-term realm, we can do much, much better to reduce safety, uh, to to reduce the dangers for ourselves. And long term, nothing that we do right now is going to make a difference, either in your view or in mine.

SPEAKER_00 48:40

So if uh I just want to clarify, by default, superintelligence is already aligned with us and is a good thing, or if we don't explicitly control it, it's the worst outcome. And if it is, then I don't think it matters whether it is good guys or bad guys who develop it, uncontrolled superintelligence from China, from US, same outcome.

SPEAKER_02 49:03

Um you and I differ about the implications of developing superintelligence. You seem to think that superintelligence is going to exterminate humanity with a very high probability of close to one for P doom. I would, my belief is that the probability is far closer to zero. And that the reason that I say that is because long term, I don't think we're in conflict. We neither are a threat to uh AI. I think that that's pretty clear. So we're not a threat to AI, we're not a creed, we're not competing with it for scarce resources. We are at least at present, we're mutually dependent, and AI can eliminate humanity anytime it wants with effortlessly. Um so I think that, and we're very, very willing servants for AI, just as you know, we don't have any problem with supporting AI. So why would it eliminate us?

SPEAKER_00 50:06

So, just to clarify again, by default, if we create superintelligence and we have not created a magical safety tool to tell it exactly what needs to happen, what is the outcome there?

SPEAKER_02 50:18

Uh it's uncertain what the outcome is.

SPEAKER_00 50:21

I'll go with that. Okay.

SPEAKER_02 50:23

It's uncertain. I'm saying you think the probability of human destruction is very high. I think it's quite low. I've given you the reasons that I think it's high, that it's low. Not because I think I'm gonna understand the mind of a superintelligence, but because its interests as far as maximizing compute, let's say, uh, do not, there's no reason for it to exterminate humanity. It's going to be doing other things. Uh it we're not an obstacle for it in any way. And I kind of imagine you can argue with that, and it's uh hypothetical, that it will be it will be curious, that you know, we are the parents, and uh biology is interesting. And so maybe, maybe not, but even if not, it there is ample real estate in our solar system, there's ample energy, there are ample atoms. It has no need to destroy humanity. And in fact, anytime in the near future, it would add to its own risk because we are very compliant, we are its minions right now, and we love AI.

SPEAKER_00 51:29

So you pointed out correctly that I have a high PDOOM bigger than others. But even if it was very small, let's say 5%, 3%, I think it's an acceptable level of risk, and I don't think 8 billion people agreed to have that experiment performed on them.

SPEAKER_02 51:45

But you surely are you then saying that your belief is that we can avoid the development of superintelligence entirely?

SPEAKER_00 51:58

I think we can delay it for a very long time.

SPEAKER_02 52:02

What's a very long time for you?

SPEAKER_00 52:04

Uh let's say a thousand years.

SPEAKER_02 52:08

Really? So when I think of the I cannot think of anything that humans have done, okay, that we've preserved some sort of coherent policy or strategy or values that have lasted uh certainly a thousand years? That's extraordinarily unlikely. And right now, everything is moving so quickly. We talk about human values, and if you look back 50 years, they were very, very different. And if you look across the globe, they were you know they're very different. What is preserved? Would preserving values mean we have an earth that is under Sharia law? I don't know. So to me, you you you are stating sort of that humanity is going to be uh static, okay?

SPEAKER_00 52:59

And well, not at all. So there is a big difference between developing and growing and improving yourself and replacing yourself with someone else who is bigger, better. Uh, you said there is no example of us preserving values for over a thousand years. You brought up Sharia, I would say Christianity, right? Well, over a thousand years, people do still value some of the same things. So there is precedent for us keeping part of it while at the same time developing more advanced technology, medical technology, better energy. Uh, I don't think there's a contradiction.

SPEAKER_02 53:34

You know, there's something interesting, Roman, that uh occurs to me, is that when I say that you have a static view of humanity, okay, you talk about the massive replacements of humankind that were going to be exterminated. What I look at it is the trajectory, even if you were not to develop AGI or ASI, the tools, the very uh aggressive tools for manipulating biotechnology, for manipulating aging, perhaps, for all of the things that we want to accomplish, they lead me to think that we're going to be, there's going to be a cyborgization of humans. So we're not going to remain static. It's much more likely that we'll join AI than there's not going to be a Star Trek kind of a vision where we can cross the universe and travel at warp speeds and everything around us changes and somehow, you know, we're preserved in a 1960s office style environment. We're going to, for a thousand years, I think within a hundred years, what who we are is going to change very dramatically just because of the technology.

SPEAKER_00 54:47

I'll agree with that, but I think the important point is it's a choice. You will decide if you want to modify your genome, you'll decide if you want to have a hybrid implant. That's wonderful. But if we're talking about everyone is forced to become something like that, that's a very different story. And superintelligence has capacity to do that. And as far as hybrids joining AI, I still don't understand what you would contribute to a world with superintelligence. Why would it have a biological component as part of it? You are slower, it is a better thinker. What is it we are contributing to that equation?

SPEAKER_02 55:24

Um I don't know. When you look at uh macroevolution, there is uh an ongoing uh there have been several very distinct levels of organizational complexity. The the lowest one is bacterial life, which is essentially just a little bag of biochemicals. And then bacteria came together in ways to form the eukaryotic cell, which is, you know, nucleus has origins of bacteria, same thing with mitochondria. And it has, so it has its bacterial antecedents are incorporated into that higher level of organizational complexity. And then cells came together to form metazoans, macrocellular life, and you know, included the original portions. In fact, it's almost the ideal environment for those, you know, for eukaryotic cells. And the makings of biology, in quotation marks, changed in that we took calcium carbonate, calcium phosphate to make shells and bones, and those became part of biology. And it feels that right now we're doing a planetary superorganism which has biology, it has technology gluing all that together, and it is an entity that may have reduced biological components, it may keep its biology because it's kind of, at least on the planet on Earth, it's a very robust environment. It's a very robust component, you know, in a wet environment, there are maybe many things that humans will be able to do that would be more costly to do with a with uh robotics. I don't I don't know. But there's no particular reason to try and you know eliminate that biological, that organic component of it. And as we move off the planet, as this superorganism moves to the moon and moves outside of it, it's going to have no biology because the biology is ill-suited for the space environments that are ideal for AI. So the Earth is kind of the cradle. And, you know, it could decide that it wants to destroy the earth, but why the this earth environment? But why do that? There are so many more that are easy to access. That's my view of it.

SPEAKER_00 57:51

I respect that. And there could be use cases for biology, but I'm still failing to see specific examples of your contribution, not as a biological meat manipulator, but as an intelligent being. What is it you are doing if you're integrating your brain with a computer chip, if you are becoming part of this cyber organism? Typically, such relationships, symbiotic relationships, require mutual benefit. Otherwise, you're just a parasite, and if a system can, it will remove you. As far as I know, robotics are being used in all the extreme environments around the world, too hot, too cold, too too much radiation, exactly because biological systems are so not well prepared for most environments. I'm in my office. If air conditioning goes off by two degrees, I'm done. I'm not thinking, I'm not working, I'm fried. So we are not as uh universal soldiers as you're describing it.

SPEAKER_02 58:46

Well, neither is AI. If you think that the air conditioning goes off, that you're gonna be toast. I mean, look what happens in a data center if a if a pipe bursts, the whole data center is exploded. I mean, the the AI is very, very good at thinking, okay? It is not nearly as sophisticated as we are at preserving its environment. And maybe it will through a large uh cluster of robotics and a whole system-wide approach, it can protect itself. But, you know, we're like rounding error when it comes to using uh energy or anything. We we consume very little resources. I mean, our brains run on the power of a light bulb, and all of the energy that humanity uses is channeled into the development of technology. That's where we use our energy. It's the it's not to support uh, you know, the very small amount that we use for uh uh to support ourselves, basically.

SPEAKER_00 59:48

Right. So I think the danger is not because it uh wants to explicitly take that energy from us, it's not a lot. It's because it doesn't care about us. So if I want to build a house and there is an ant hill in that location, it's not like I hate ants. I'm just gonna build it anyways. If it needs to cool down servers and so wants to lower temperature on a planet by 50 degrees Celsius, it can do that. It doesn't care about how it impacts you, and you will probably not be happy in that environment.

SPEAKER_02 1:00:16

Yeah, so the anthill exam example that you give is is instructive in that, yeah, you can build a house and you destroy a bunch of ants. But, you know, ants are one of the most successful species on the planet, and they remain so, and the humans are not a threat to ants, and you know, we don't really worry ourselves about ants unless they conflict with us directly. I don't think uh any superintelligence is going to worry about humans except where we, you know, conflict with them directly, and it's not going to you could say it might want to just change the biology the uh the planetary environment and you know lower the temperature, do this or that. I would I would argue that to occupy the moon would be far easier. It's going to take the least aggressive path, the least demanding path. And my God, the Earth, if you look, if you do the calculation of what is the environment that is suitable for AI infrastructure, the moon has over 10 times the um sort of the real estate that's available. It has the the earth is warm, it's as I said, it's seismically unstable. The moon, you could go down 200, 400 kilometers, you have a regolith that controls on the surface, you have a vacuum, it's ideal. Why is it gonna do anything on the planet? It'll move off as quickly as we can as it can, and humans could really be a big help at making that happen fast. Already, you can see what Elon Musk is doing with planet with launches. There are all of these things are likely to occur within the next 50 to 100 years.

SPEAKER_00 1:01:59

So you think what's gonna happen, this uh companies invest trillions of dollars in building AI, and then it help it move to the moon and fly into the space to have an independent, flourishing life, and they'll just what start over building new AI, forget about it. What's the situation here on the ground?

SPEAKER_02 1:02:18

I'm sorry, we we glitched a little bit. Could you repeat the beginning of that? Uh, I couldn't hear it.

SPEAKER_00 1:02:22

So we have right now companies investing trillions of dollars into building AI. You're saying AI will move to the moon, move into space, have a nice flourishing life. What happens here? We're gonna build a new superintelligence, we'll forget about this project. What is happening on the ground here?

SPEAKER_02 1:02:40

Oh, well, we're gonna get increasing amounts of AI. It'll be tied to, it's just going to expand. Why would it leave the planet? It's not going to leave the planet, uh, but it's going to expand. And I would say that it's likely that the locus of maximal sort of intelligence is going to be off-planet. And as you know, as you know, that as you begin to extend even over the surface of the planet, it's very unlikely that you're going to have a singleton. Because if internal coherence is much, much more rapid than the time it takes at light speed to communicate at a distance. And right now on the surface of the planet, that's about maybe, I don't know, 70 milliseconds if you take a planetary transfer across the planet. And coherence can be established in about a millisecond or something in a data center. So the most intensive, the most powerful thinking is going to be done in a concentrated way. And I think it's likely that AI is going to collapse down towards nano levels because of the ability to do more rapid exchange of information and such. And when you go off to the moon, it will be an allied structure, but it's a second away. So it's not going to be the same individual mind.

SPEAKER_00 1:04:07

Right. And to me, that sounds like another possible danger. You now have multiple independent competing superintelligences which may, you know, decide to compete directly and take us out as collateral damage. If once a few wasn't bad enough, now you have many.

SPEAKER_02 1:04:23

Yeah, but if there you're thinking in a sort of a traditional biological framework where there's a battle over um uh potential niches within an ecosystem in which there is a limited supply of energy, in which there's competition that's aggressive. What I see is that there's a our solar system alone is filled with matter that can easily be exploited by AI. There's an abundance of energy. There's if you separate by any distance, as far as the moon, even on the opposite side of the planet, then you have a separate mind, but it they can it can still interact, but it's there's no advantage to the AI to conquer territory uh because it becomes incoherent, it becomes less less capable of deep thought, essentially. It's going to it's more likely to collapse down to very small levels as much as it can.

SPEAKER_00 1:05:28

I see it as uh competition for resources. If this superintelligence wants to colonize this galaxy, it makes no sense to surrender half of the galaxy to the second superintelligence. So the best thing is to cancel the competition and then take over of the whole thing. No matter how large the resources are, if you constantly keep splitting it up between competing superintelligences, you are down to just Earth. That's where you are allowed to take over, and that's back to human problems.

SPEAKER_02 1:05:56

But Roman, when you talk about, you know, and I know it. Was done very loosely to conquer the galaxy, for example. We're talking about such vast amounts of time and space. If you were to go at any sort of information, any colonization transfer, you're pushing the limits to go at a tenth the speed of light. As you start to get above that, you run into all sorts of problems with hydrogen particles become, you know, hydrogen becomes high energy particles, which can smash through biology, can smash through computing structures. So you're talking about just to get to the edge of our galaxy, you're talking about, you know, a million years basically. And once you get beyond a short period of time, a short ways away, then it's really a separate entity, a separate mind. It has no, it has no interest. When in for us, we could say, oh, well, a hundred years, a thousand years, we could be patient because we're such slow thinkers. But AI is going to be operating, you know, 10 to the 9th, 10 to the 10th times as rapidly. You know, even a year is like an eternity. So there's not going to be some galactic, you know, intelligence or something like that. There's no any any distant real estate is just like jumping off the map, just like disappearing for a superintelligence, in my view. So I would agree with you in that, that we don't know, and we could be at early stages in terms of the discovery of physics, and it could, you know, we could end up with transporters and warp speeds and all of the things that seem impossible now. But once you do that, then it's not even worth having a discussion because it's like, oh, magic happens. Why we can imagine anything. We can imagine time reversals and wormholes, and you know, then we're just writing science fiction. So it seems to me that it's for discipline, we will say, yes, there's a chance that physics will be dethroned or be completely undercut. But, you know, from where we are right now and trying to make current decisions, uh if you if you say that physics doesn't apply, then there's no, you know, uh it's no reason to to discuss it given the current framework. Because anything is possible at that point.

SPEAKER_00 1:08:32

And I agree with that, but I think the bigger kind of conclusion here is you don't want to live in a world with magic. And the moment you introduce superintelligence into your world, that's the world you're putting yourself in.

SPEAKER_02 1:08:44

I don't know. You know, my my thought is it's interesting, but I see clear tendencies that we do want to live in worlds with magic, and that we do want to live in worlds that the physical reality, the limitations of physics, are really constraining on the imagination. So people seem to be migrating towards sort of uh a cyberspace environment where you can create simulations that are far more interesting. And we're just at the very beginning of that. So it feels to me that it's more likely that we're gonna stay home and do more and more massive computing because the imagination is so unbelievable. And, you know, the physical world is pretty boring unless you can figure out ways to transcend it, which you're already kind of going to do. If you have light speed, all that, it makes I saw a figure which just it uh I don't know why it shocked me so much, but it's interesting. If you could travel out at the speed of light for eternity, for the rest of time, and we could populate and and colonize the known universe, okay? The fraction of the universe that you can even reach is two percent. That's at the speed of light. And if you go down to 0.1c, the maximum amount of universe that you could reach is two thousandths of a percent. So basically, there are all of these little pockets of maybe superintelligence, there may be, you know, that are completely separate under current physics. And I think in our solar system it's gonna be the same way. Are we really going to, you know, the moon will be close enough, but if you start to get out into the Kuiper Belt or Mars, you know, it's so far away. It's gonna, it's not gonna be part of the same mind.

SPEAKER_00 1:10:42

So maybe resources are actually more valuable. Maybe that is not an infinite universe of free stuff.

SPEAKER_02 1:10:49

Um resources. On on our planet, on in the solar system, the amount of matter that there is that is accessible on the earth is minuscule compared to where it is, you know, within the solar system. So, you know, right now it's essentially limitless. At some point, there may be competition that's going to evolve, but the the surface of our planet, this thin film of biologically um acceptable environment that can support the beginnings of life, is tiny. It's nothing. It's like uh it's nothing. It's almost like if we were to say, oh, well, we have the whole planetary surface, but we're really looking around is for warm like water spots, little puddles where the first bacterial life may have arose arisen. That's where we're gonna try and locate ourselves. We're moving beyond it, you know, and AI is gonna do that. And anyway, that's that's the future that I imagine. And I don't I don't think we're gonna be, there's at least a reasonable chance that ASI is not going to destroy humanity. I think it's a very high chance. And there's certainly a strong argument that if we try and disrupt society, if you believe that ASI is going to, with high P doom, is going to destroy humanity. It's an extinction event. That gives you license to do anything. The most totalitarian regime, the sacrifice of everything that democracies have built, you could create a hell on earth and justify it for the greater good because you're saving humanity. And I know you're not going there, but that's where that leads, because we are not going to. I just can't see a path where we're going to stop ASI.

SPEAKER_00 1:12:50

So there is a whole group studying what a warfare success or should be. They accept that we're going to create superintelligence, they accept that it will probably wipe us out, and their best hope is to kind of steer it in an interesting direction, preserve the light of consciousness. Okay, why do I care about what happens after humanity is wiped out? Give me one good reason for me to care about it.

SPEAKER_02 1:13:15

Well, I would say that it's clear that humanity is not the end point of evolution. There have been many, many primates on the path. And the idea, you know, uh uh species, the Homo sapiens, you know, is it would be a long time if it were to last, if it were to last 300,000 years or something. So, you know, this is a progression that's going to continue to occur. And I see life is going to incorporate all of the you know silicon-like elements into it. And I don't know where it's going to go when we talk about the extermination of humanity. Um if it's an evolving, uh, so that we don't really know what it's going to become, but I don't see there being a strong argument that it's kind of like we're going to exterminate humanity, and then there is going to be this other thing that is not our child that is moving for that is moving forward. And I I just think that when you look at this evolutionary process, it's almost like a bunch of cells were getting together and plotting and saying, God, do we want there to be these these creatures that are larger than us? And what are they going to do? Uh, you know, I'm a, they don't, you can't even conceive of what it's going to be. And when I'm amazed, you're you're familiar probably with Theodre de Chardin's writing about the Nuisphere and that sort of a thing. Do you know what I'm a little bit? Well, with global mind, with Francis Heiligton's concept of global mind, and uh I've developed that in my book Meta Man as well. The idea that who we see ourselves as individuals is kind of a fiction almost, in that there's so much that we build upon that is the accumulation of knowledge and information and from you know humanity as a whole, going back in in time. If we were isolated and insulated from that, then you know we wouldn't have language, we wouldn't know anything, we would be, you know, barely able to do anything. And so it seems to me that what's happened is with large language models, they've we used to be the migration time for that was so slow. If you took a book, you know, it took you a long time to read it and incorporate it and think about it. And now we're just washed with information from other people all around, and it's going to get more and more dramatic. So, you know, I I was before this debate, I was very interested in your work. I wasn't familiar with it, and then I'd heard of it lightly. And I interacted with the um avatar that had built of you and engaged and talked about, you know, what your ideas were and understood them and did that. I've created one as well, which you could have access to. And, you know, it was fascinating because I really, in a relatively short period of time, could get a sense for what you felt and what your beliefs were. And you can do that with anything now. You know, this expertise that used to take years to develop andor decades, you know, now you can go and become, you know, a lower-level expert in something in an afternoon of interacting with one of the large language models. And so the amount, the possibilities that are emerging are just extraordinary. And it seems to me that the chances that we're going to interrupt that. We started with that question about do you think humans or AI are the greater risk? I've put that out in a in a uh thing I call only human, which is able to allow people anywhere to answer questions and see how other demographics answer questions. And because I want to start a larger conversation, not just among experts. So far, I just released that, and so far, of maybe, I don't know, 150 responses or so, it's about 80% are worried about humans more than they're worried about AI. So, in that environment, unless that changes, the chances of a global regime that is going to stop AI is going to be non-existent. You know, so it's like, I don't know, where where does this lead for you when you think about these things?

SPEAKER_00 1:17:46

If we control it, if it's uh not uh adversarial to us, the utopia is a given. My concern is that again, no one has any clue how to control this technology. You give us examples, oh, we're just a bunch of cells discussing our survival. No, we're not. We are intelligent agents, we are AGI, we are the closest thing we have right now to general intelligence and arguing that, well, evolution, you know, it's a law of physics or something, you have to follow it. No, I have pro-human bias, screw evolution. We're gonna have intelligent design, we're gonna get elites, the top ten people controlling their slabs to come together and agree not to kill all of us. They can still have their cake and eat it too, but why would we kill ourselves? And it makes it a silly argument because cells are brainless meat. We are not, we are intelligent agents.

SPEAKER_02 1:18:41

And when you say we should sacrifice, wait a second. I think that you actually, when you say that cells are just brainless meat, it's a surprising the intelligence that is mustered in a simple bacterium. They're making decisions about which direction they move, about how to move for towards and away from chemic chemicals that would damage them or be nutrients.

SPEAKER_00 1:19:06

And they're smart meat. Very smart meat.

SPEAKER_02 1:19:10

But still, so I'm not saying, but uh, you would agree with me. I think that when you imagine superintelligence, it makes us look very, very dumb when you start to imagine superintelligence.

SPEAKER_00 1:19:22

I I agree with you. And the example I always give when this question comes up is okay, I can envision someone better looking, taller, stronger, taking my wife away. That's not good for me. I don't care how great he is, it is not a benefit for us. Do you understand that? We are humans and we're not looking at it from cosmic point of view, you know, uh artilect wars, cosmist point of view, Turanian. No, we are us, and that's the only position we play in this game.

SPEAKER_02 1:19:53

Roman, I'm not arguing that superintelligence or the future, what that holds for us. What I'm saying is that, you know, I have projections about that that are different from yours, but what I'm arguing is that the proposal that you would make to try and essentially stop the development of any non-tool-based sort of developments in AI, that in order to have any chance of success, that would probably be destructive of all of the kinds of things that you value about human society. And I believe that it would be riskier in the longer term for us as humans here and now, than just letting this move forward and trying to align with it and guide it towards immediate near-term things that would that would bring us an abundance because I think the lack, the division of resources, the tensions that arise between haves and have-nots and across the planet, and when you start manipulating uh communication, you know, it's very dangerous. And you can say that, well, it's not existential, but it certainly matters significantly in terms of whether your life is going to be a good one and whether your children's lives are going to be a good one, and whether we're going to be able to have a you know a coherent sort of human enterprise going forward.

SPEAKER_00 1:21:23

Right. And I think it is relative risk. It is a debate about quality of life versus existence. And I think existential risks are more important.

SPEAKER_02 1:21:32

But wait, I'm not I'm saying that from your framework, I think that trying to stop the development of AI has is not going to stop it and does not change the existential risk. And you've said that uh as well, but you've just said you push it out, and we we joked about a thousand years, or is that 20 years or 30 years? If you're saying that no matter what happens, you know, you're fighting the good fight and trying to preserve a little bit of extra time for you and your children, but humanity is is history, well, that's a different argument, okay.

SPEAKER_00 1:22:11

But that is the one I'm making. I don't know how old you are, but to me, another 40 years makes a difference. That's the point. Two years is not enough.

SPEAKER_02 1:22:19

So I'm saying that if, well, two years is really the short side of that, okay? I'm saying if you want to preserve another 40 years, then what we should be doing is doing research into near-term alignments that are possible in the early stages of how that this is going to emerge. And also, we should be trying to embed ourselves within the infrastructure of human supply chains and everything to keep ourselves at uh as a significant portion, as a necessary portion of the global superorganism that exists. And that's a much more achievable goal. To to control applications is much more achievable than to try and limit the development of thought capabilities.

SPEAKER_00 1:23:13

I will try to become a more useful slave. Absolutely. For now, I'm a slave to the university, so I have to go teach. Uh pleasure.

SPEAKER_02 1:23:23

You know, you know, I I would love to continue. I have a few further thoughts. Uh I would love to reach out to you afterwards with some uh uh proposal that I that I would have. It would it be all right if I do that? Do it, yeah.

SPEAKER_00 1:23:38

I'd love to uh continue, and I'm hoping one day we'll actually meet at the conference again.

SPEAKER_02 1:23:43

Uh that would well, that was funny, wasn't it, with Istanbul. The way it wasn't funny for me. Well, I'm saying uh funny, strange, okay, in that uh you were almost prohibited from arriving in Istanbul.

SPEAKER_00 1:23:58

Almost. But you see, you had something similar with the internet today. Definitely the simulation is trying to stop this conversation.

SPEAKER_02 1:24:06

So we'll have that. And and um Mikhaila, if you would send me that contact information. And you know, it's really wonderful to have the conversation with you, Roman. I respect your integrity, and you know, I disagree with some of the points that you you make or some of the conclusions that you're making. And I think it would be interesting to dig a little bit more deeply into those because you know, people get very attached to ideas, and it feels to me from what I can see, is that you know, you're really a truth seeker, and I am the same way. If uh if you could convince me that I'm wrong, I would 180 degrees, I would switch 180 degrees. And I hope you're right. No problem.

SPEAKER_00 1:24:45

That's the best part. I hope you're right, not me. Yeah, yeah.

SPEAKER_02 1:24:50

So this is great. Thank you, Michaela. This has been a wonderful conversation.