The Button that Might Destroy the World
Imagine a button. Maybe it is big and red and labelled “danger.” Now imagine that there’s decent reasons to believe that there is a very slight chance, numerically somewhere around 0.1% but it’s impossible to know so accurately, that if you press the button, the world will be destroyed. Now imagine that if you do push the button, you have a much more reasonable chance of making a lot of money and getting significantly more opportunity in life. Not pressing it runs a risk that you have to undergo significant hardship and may lose a lot of your financial security and life’s work. Would you press the button?
It would be hard for me not to push the button. More importantly, I think there are many many people who would push the button. 0.1% is quite scary for something so negative, but one imagines that it can be lowered and most people have a point where the personal benefits would outweigh it. 0.001 or 0.00001 seem quite reasonable and compatible with a normal moral person. It is understandable. Change the chances of the risks and benefits and it becomes harder to resist. Maybe it doesn’t destroy the world, maybe it just kills a lot.
The problem of course is that if enough people hit the button enough times, we all lose. So it’s very important that the button does not exist.
The button can be many things. It might be deploying an AI system that has a tiny minuscule probability of self-replicating and destroying the world, but could prove absolutely crucial for the competitive survival of a major tech company. It could be the authorization by an administrator of a bio lab which is only slightly unsafe, but an experiment goes wrong from careless procedures and a bad virus gets made. It could be the choice of the United States to explode an atomic bomb to win WWII without one hundred percent confidence that the process would have unforeseen catastrophic effects, a risk with all new technology.
Buttons are rare, but we are working very hard to make more. As humanity’s technological powers increase, we increase our power to kill ourselves. This is an unavoidable consequence of increasing our power at all: mastering reality grants the ability to do both good and evil. The good should not be ignored; bioengineering, nuclear technology, and AI systems have immense benefits to offer even if they are linked with risk. We are expanding our practices around safety - the cooperation for nuclear disarmament would not have been possible a century ago and oversight causes us to prefer safer branches of technology - but our desire to move forward is still creating more opportunity for danger than assurance of safety.
There is an alarmingly small amount of work being done to reduce major risks. A common opinion is that humanity is now much wiser and can be trusted with its deadly tools. An intentionally engineered pandemic hasn’t happened, so it probably won’t. The lack of nuclear war in all the time since Trinity is considered evidence that we are safe. But there is a lot more time ahead than that. Most of the work being done is trying to disable the buttons that do exist, prevent more buttons from being made, and make sure buttons are mostly in the hands of trustworthy actors. But we are still making buttons and it is hard to be certain that they will be fully detachable from any possible personal motivations to press them.
We should work too on actually becoming wiser. Actually making people less likely to press the button. If we continue to grow in power, we cannot trust that the power will only be held by the trustworthy. Keeping weapons in the hands of governments and corporations only buys time for the larger project of improving wisdom, because leaks are inevitable in the longterm. At some point, some person, motivation, and opportunity will arise. If we can create a civilization where every individual can be trusted with the power to destroy the world then our full possibility can be unleashed for the good. If we cannot our options are some mix of progress, stagnation, and doom.
This will not be easy. Most approaches to safety correctly begin from the fact that man is fallible, greedy, and stupid. Safe systems are designed to work even in the case of these failures. But in addition to being fallible, greedy, and stupid, man has the potential for wisdom and perfection. If we want to continue to grow and change, we ignore our potential at our own peril.
There are a few possible approaches to transforming button-pressers. One is to significantly improve moral values and self-control. I personally wish I had enough self-sacrifice and courage so that I could be sure I would not press the button. The greed and selfishness that motivates a button press will hopefully fade in a more enlightened people. Another is decreasing the motivations. If people are already happy and satisfied, if they do not fear falling or failing because they know others will save them, the button’s allure fades. The social stigma of pressing a button is not worth any possible gain or advantage. And decreasing deviance or unpredictable behaviors is also likely important. Most people do not want to destroy the world, but not all. In the grips of severe mental illness or zealotry, some people will horrible evil, even if just for a moment. Such people are kept away from buttons due to the hierarchal nature of society, but this may not always be the case. Eventually, I hope that we no longer have to fear these ailments and their sufferers are healed.
Until this can happen, which might be very long indeed, the more practical safety program putting power in the hands of systems rather than people should absolutely be followed. But we should use the time we gain to reach the more complete and safer solution.