Scary AI Is More “Fantasia” Than “Terminator”

When Nate Soares psychoanalyzes himself, he sounds less Freudian than Spockian. As a boy, he’d see people acting in ways he never would “unless I was acting maliciously,” the former Google software engineer, who now heads the non-profit Machine Intelligence Research Institute, reflected in a blog post last year. “I would automatically, on a gut level, assume that the other person must be malicious.” It’s a habit anyone who’s read or heard David Foster Wallace’s “This is Water” speech will recognize.

Later Soares realized this folly when his “models of other people” became “sufficiently diverse”—which isn’t to say they’re foolproof, he wrote in the same post. “I’m probably still prone to occasionally believing someone is malicious when they’re merely different than me, especially in cases where they act similarly to me most of the time (thereby fooling my gut-level person-modeler into modeling them too much like myself).” He suspected that this “failure mode” links up with the “typical mind” fallacy, and is “therefore,” he concluded, “difficult to beat in general.”

Beating biases is one of Soares’ core concerns—“I care a lot about having accurate beliefs,” he says—and one of Neil deGrasse Tyson that keeping a super intelligent machine “in a box,” disconnected from the Internet, was a lousy safeguard for a misaligned AI and Steven Pinker’s misconceptions about AI safety. Donors find the Institute’s mission worthwhile: In December 2017, 341 of them it $2.5 million.

You're reading a preview, sign up to read more.

More from Nautilus

Nautilus9 min read
Tantalizing Creatures with Male and Female Genes: Gyandromorphs overturn traditional theories of sexual development.
As they often do after a rainstorm, butterflies had gathered around puddles on Pigeon Mountain in northwest Georgia. Nets in hand, James Adams and his friend Irving Finkelstein watched the insects lapping up salts and proteins dissolved in the muddy
Nautilus7 min read
Evolution Is Really Not That Into Sex: Plants and animals have reproduced without sex for eons. So why did nature bother?
What is sex for? When I regularly queried students at the beginning of my course in evolutionary biology, most responded that it was for reproducing. Reasonable enough, but wrong. In fact, lots of living things reproduce without sex: Asexual reproduc
Nautilus9 min readSociety
Nurture Alone Can’t Explain Male Aggression
A young bank teller is shot dead during a robbery. The robber flees in a stolen van and is chased down the motorway by a convoy of police cars. Careening through traffic, the robber runs several cars off the road and clips several more. Eventually, t