AI Safety is, brutally put, the quest to avoid that superintelligent robots kill us all one day.
Why would they kill us? This could happen for at least two reasons. Either we might totally mess up the goal we give them: For example, if we task a superintelligent AI to make all humans happy, it might behave benignly for a while until it achieves sufficient power to kill everyone. When every human is dead, no human is not happy, and the goal is perfected. Though it might sound stupid, it's generally hard to predict how a superintelligence will interpret a goal.
Another possible reason is that almost regardless the goal, resources will help a lot in achieving it. And humans control (and consist of) a lot of resources.
Stuart Russell, top professor in AI and author of the famous textbook Artificial Intelligence: A Modern Approach , elaborates on these points:
AI Safety does not mean that we shouldn't build superintelligent machines, only that we should spend a fair deal of effort to avoid the potentially very bad outcomes.
Learn how to construct AIs that do what we want them to do.
For this, we have to solve the AI Control Problem, which is plenty harder than one would first expect. Nonetheless, this is the most explored venue for AI Safety. See Bostrom's book Superintelligence for details.
Structure society in a way that no agent, especially no super-AI, will have incentives to do damage ("crime shouldn't pay").
Whether this path possible depends on how fast super-AI's can self-improve: If the improvement is too fast, a single agent may become so powerful that no law or societal structure can stop it. In this case, solving the AI Control Problem would be safer.
If self-improvement is a bit slower, a mixed economy of humans and AI's may arise (see Hanson Economics Of The Singularity). In the mixed economy, humans will have a much better chance to prosper if they can properly control super AI's to work for their benefit. This is a second good reason to solve the AI Control Problem.
Some AI Safety resources:
A bit longer and more technical talk by Nick Bostrom given at Google:
Omohundro (2008), The Basic AI Drives provides good intuition for what drives we might expect from any sufficiently intelligent system.
Chalmers (2010), The Singularity: A Philosophical Analysis argues for why superintelligence is likely to be developed soon, and explores some associated challenges.
Sotala and Yampolskiy (2015), Responses to Catastrophic AGI Risk: a Survey summarises a large set of potential AI Safety approaches, and gives plenty of references.
Machine Intelligence Research Institute is an independent research institute that does foundational research on the AI Safety problem.
Future of Life Institute funds research in AI Safety, among other things.
Future of Humanity Institute is an Oxford University institute focusing on existential risks in general, with strong emphasis on AI Safety.
Back to tomeveritt.se