Watson's Mistakes

James Barrat has identified IBM's Watson as "the first machine to make us wonder if it understands" (224).  But many critics, including many computer scientists, would say that this is an illusion.  There is no reason to wonder whether Watson understands what it is doing when it plays the Jeopardy game, because Watson is little more than a fancy search engine.

Watson has downloaded millions of documents, including encyclopedias, books, newspapers, and the whole of Wikipedia.  When it is presented with a question or a Jeopardy clue, it analyses the language and does a statistical word-search across its documents for matching language, comparable to the way the Google search engine scans the Internet for us when we type in some words.  Watson must then find a match between the significant phrases in the original question and phrases from its search.  From this, Watson can produce an answer that looks like the product of intelligent thinking, comparable to the thinking of Ken Jennings and Brad Rutter when they play Jeopardy.  But surely thinking and searching are not the same thing.

But if we interpret the success of Jennings and Rutter in playing Jeopardy as evidence of their human thinking, why shouldn't we interpret the success of Watson in defeating them in Jeopardy as evidence of his machine thinking?

After all, even Jennings said that he felt that Watson was thinking like a human competitor:

"The computer's techniques for unraveling Jeopardy clues sounded just like mine.  That machine zeroes in on key words in a clue, then combs its memory (in Watson's case, a fifteen-terabyte data bank of human knowledge) for clusters of associations with those words.  It rigorously checks the top hits against all the contextual information it can muster: the category name; the kind of answer being sought; the time, place, and gender hinted at in the clue; and so on.  And when it feels 'sure' enough, it decides to buzz.  This is all an instant, intuitive process for a human Jeopardy player, but I felt convinced that under the hood my brain was doing more or less the same thing." ("My Puny Human Brain," Slate, February 26, 2011)

Or was Jennings caught up in the same illusion that captivated so many other people who observed Watson at work?  Can we dispel this illusion by looking at Watson's mistakes in the game that reveal his mechanical stupidity and failure to understand anything? 

Consider the following three examples of Watson's mistakes.

1.  "Olympic Oddities" in the Jeopardy Round for $1,000.

It was the anatomical oddity of U.S. gymnast George Eyser who won a gold medal on the parallel bars in 1904.

Jennings:  What is he only had one hand?

Watson:  What is leg?

Correct answer: What is he's missing a leg?

Triple Stumper (all three contestants failed to answer correctly)


2. "The Art of the Steal" in the Double Jeopardy Round for $1,600.

In May 2010, 5 paintings worth $125 million by Braque, Matisse, & 3 others left Paris' Museum of this art period.

Watson: What is Picasso?

Jennings:  What is cubism?

Rutter:  What is impressionism?

Correct answer: What is modern art?

Triple Stumper.


3.  "U.S. Cities" in Final Jeopardy Round.

The largest airport is named for a World War II hero, its second largest for a World War II battle.

Jennings:  What is Chicago?

Rutter:  What is Chicago?

Watson:  What is Toronto?????


Notice first that two of these three were Triple Stumpers.  So the two human contestants were no better than Watson in unraveling two of these clues.

If you go to the Wikipedia page for Eyser, you will see that he had lost his left leg as a child, and it was replaced with a wooden leg.  Seeing this, Watson answered: "What is leg?"  Alex Trebek said "Yes."  But then a judge stopped the game.  After a five-minute discussion, the judges decided this was the wrong answer, because "leg" was not the "anatomical oddity," but the fact that he was missing a leg.  If Watson had answered--"What is a wooden leg?"--that might have provoked another discussion as to whether that was the correct answer.

We can see that Jennings knew by common sense that for a gymnast doing parallel bars, agile use of hands, arms, and legs is normal, and so missing one of these would be an "oddity."  But he could only guess which one was missing.  In the few seconds he had to think about it, Jennings did not have time to figure out that artificial arms or hands in 1904 would have been too crude for a gold medal performance, but a wooden leg might have been less disabling.

Watson knew that "anatomical" could include leg.  But the Wikipedia page does not include the word "oddity."  Any human being reading the Wikipedia page would immediately identify missing a natural leg and having a wooden leg as an "oddity" for an Olympic gymnast.  This is part of what computer scientists have called "common sense knowledge"--the massive accumulation of informal knowledge that human beings acquire by experience without any explicit instruction, but which computers do not have.

Although it is a daunting project, providing computers with human common sense knowledge is in principle possible.  After all, if Watson had had the information in its data base that "missing a leg is an oddity for an Olympic gymnast," Watson could have answered correctly.  Beginning in 1984, Doug Lenat and his colleagues at Cycorp has been building an expert system--CYC (for encyclopedic)--that codes common sense knowledge to provide machines with an ability to understand the unspoken assumptions underlying human ideas and reasoning.  Lenat wants CYC to master hundreds of millions of things that a typical person knows about the world.  For example, consider the common sense knowledge that birds can generally fly, but not ostriches and penguins, and not dead birds, and not birds with their feet in cement, and not . . . .  It is hard but not impossible to formalize all such common sense knowledge.

Watson's mistake about the art theft clue shows how Jeopardy clues are often confusing in that it's hard to interpret what kind of answer is being sought.  In this case, the clue wasn't seeking the name of an artist or an art period as such, but the name of a museum--the Musee d'Art Moderne, the Museum of Modern Art, in Paris.  Jennings and Rutter couldn't come up with the right answer.  Watson's first choice--"Picasso"--was also wrong.  But Watson did have the right answer--"Modern Art"--as his third choice!

Watson was widely ridiculed for his mistaken answer "Toronto" under the "U.S. Cities" category.  This machine is so dumb that he thinks Toronto is a U.S. city!  But notice that Watson put multiple question marks after his answer to indicate a low level of confidence.  The confidence level for "Toronto" was only 14%.  "Chicago" was his second ranked answer at 11% confidence.

Why did Watson guess that Toronto was an American city?  There are lots of small towns in the United States named Toronto, but none are large cities with large airports.  In doing his statistical analysis, Watson might have noticed that the United States is often called America, and Toronto, Ontario, is a North American city that has a baseball team (the Blue Jays) that is in the American League.

This all shows how Watson can become so confused that he cannot confidently find the right answer.  But, of course, this also happens to the human contestants in Jeopardy. 

Searle would say that we just know that human beings can think, and machine's can't!

But what if, after the game was over, it was announced that the game was actually a Turing Test--that Ken Jennings was actually a robot designed to look like Jennings, and that the robot's intelligence was Watson's?  Searle would say that even if this had happened, it would not have proven that Watson can think, because a machine can pass the Turing Test without really understanding anything.  The problem with this argument is that it throws us into solipsism, because it would mean that we cannot even be sure that human beings understand anything based on what we observe of their behavior.

Is IBM's Watson a Machine That Thinks?

“I for one welcome our new computer overlords.” 

That is what Ken Jennings said when he was defeated in the television game show Jeopardy by Watson, an artificially intelligent machine built by IBM.

There's a short video on this.  There's also a longer PBS NOVA documentary on this.  You can see all of the clues and answers in this series of games here, here, and here.  To see the correct answer and which contestant got the correct answer, you must click on the monetary value for each clue.

Jeopardy is a long-running television game show, in which contestants are given clues that are answers to questions, and they must guess the question.  The clues are put under different categories.  Each clue is assigned a monetary value, with the harder clues having higher values.  Once the clue is displayed on a board and read by the host, the first of the three contestants to press a buzzer is given the chance to answer.   A correct answer is rewarded, and an incorrect answer is punished.

Here are three examples.  Under the category “Literary Character APB (All Points Bulletin),” a $600 clue is “Wanted for general evilness; last seen at the Tower of Barad-Dur; it’s a giant eye, folks, kinda hard to miss.”  The correct answer is “What is Sauron?”

Under the category “Dialing for Dialects,” a clue for $800 is “While Maltese borrows many words from Italian, it developed from a dialect of this Semitic language.”  The correct answer is “What is Arabic?”

Under the category “Church” and “State,” a clue for $1,600 is “It can mean to develop gradually in the mind or to carry during pregnancy.”  The correct answer is “What is gestate?”

Watson correctly answered these clues and many more in playing the game against Ken Jennings and Brad Rutter.  Jennings was the all-time champion, having won 74 Jeopardy Games in a row, winning prize money of $2,520, 700.  Rutter was the all-time money winner, winning $4,355,102.

Winning Jeopardy requires not just knowledge, speed, and accuracy but also game strategy.  The game has three contestants who play three rounds--Jeopardy, Double Jeopardy, and Final Jeopardy.  In the first round of Jeopardy, there are 30 clues available, classified under six categories. Each of the six categories contains five clues which are valued at $100, $200, $300, $400 and $500. The higher values are for clues that are more difficult to unravel.  In Double Jeopardy, there are also 30 clues in six categories, but the values are doubled.

The game begins with the winner of the previous game selecting a clue.  The clue is then displayed on the board and then read by the host.  When the host has finished reading the clue, the first contestant to press a buzzer has the chance to answer.  A correct answer wins the value of the clue.  An incorrect answer is punished by having that value deducted from the contestant's score.  So contestants must strategize in deciding whether their confidence in their answer is strong enough to run the risk of getting it wrong.  The contestant who gets the correct answer selects the next clue.  When a contestant gives an incorrect answer, the first of the other two contestants to press the buzzer has the chance to answer.

One of Jeopardy's hidden clues and two of Double Jeopardy's clues are called Daily Doubles.  With a Daily Double, a contestant can bet some amount of the contestant's winnings (from $5 to all of it).  This allows the contestants to double their winnings by betting the whole of their winnings so far.  But if they give an incorrect answer, they lose whatever they have betted.

Final Jeopardy  is the last clue of the game.  As with Double Jeopardy, contestants can bet all or any part of their winnings, but all three contestants participate.  Each writes an answer and a bet, without seeing what the other two are writing.  Often, the outcome of the whole game depends on their strategy in Final Jeopardy.  The contestant with the largest winnings at the end is the winner.

Notice the many subtle, strategic decisions that must be made.  Contestants must calculate their bets in Daily Doubles and Final Jeopardy.  They must estimate their chances on clues they have not seen.  The must weigh the risk of an incorrect answer before they decide to press the buzzer, and they have only seconds to make this decision.  They must also anticipate the decisions of their opponents, particularly in Final Jeopardy where one bet can win or lose the whole game.

IBM had built the chess-playing machine Deep Blue that defeated Gary Kasparov, the reigning world champion in chess, in 1997.  This was impressive, but it did not show that AI machines are capable of general intelligence and flexible judgment comparable to that of human beings.  Chess is a restricted domain with clear rules and a clear objective (capturing the King).  By contrast, success in playing Jeopardy! requires general knowledge of history, culture, literature, and science.  It also depends on flexibility in interpreting puns, metaphors, and other nuances of language. 

The scientists decided that if they could build an AI machine that could defeat a Jeopardy! champion like Ken Jennings, this would show that artificial intelligence was finally moving towards general intelligence like that of human beings.  In 2011, Watson did indeed defeat Jennings and Rutter in playing the game.

In that game, Watson did not have the capacities for hearing speech or reading texts, but now it has those capacities.  Scientists at IBM want Watson to read massive quantities of medical literature so that it can become a medical diagnostician.  It might also read legal texts, so that it can become a legal consultant.

In much of the older AI research, it was assumed that intelligence could be reduced to facts and rules—accumulate lots of factual data and rules for inferring conclusions from those facts.  But, in fact, much of what we identify as intelligence is intuitive judgment that is acquired by learning from experience, which cannot be completely reduced to rules and facts. 

Watson’s great achievement is that it can learn on its own.  It has accumulated massive quantities of data from encyclopedias, novels, newspapers, and all of Wikipedia—the equivalent of thousands of books.  Then it surveys this data looking for patterns.  It has also surveyed 10,000 of old Jeopardy questions and answers looking for patterns of success and failure.

Machine learning from examples allows machines to acquire knowledge that cannot be reduced to facts and rules.  For example, the skills for speech recognition and reading texts cannot be achieved through a simple set of rules.  How do we recognize the letter “A”?  There are many different fonts in which this letter might be printed, and the hand-written letter differs in the hand-writing style of different writers.  But if you give an intelligent machine millions of examples of the printed and hand-written letter “A,” and the machine looks for recurrent patterns, it can learn to recognize this letter.  Similarly, speakers differ in how they pronounce letters and words, and so there is no clear set of rules for identifying spoken letters and words.  But if you give an intelligent machine millions of examples of how a certain letter or word is pronounced by different speakers, the machine can learn to identify the patterns.
 
From his experience in competing against Watson, Jennings decided that Watson was a lot like the human players of Jeopardy.  “Watson has lots in common with a top-ranked human Jeopardy player,” Jennings observed.  “It’s very smart, very fast, speaks in an uneven monotone, and has never known the touch of a woman.”

Jennings also decided that Watson’s way of solving Jeopardy puzzles was similar to his own:

“The computer’s techniques for unraveling Jeopardy clues sounded just like mine.  That machine zeroes in on key words in a clue, then combs its memory (in Watson’s case, a 15-terbyte data bank of human knowledge) for clusters of associations with those words.  It rigorously checks the top hits against all the contextual information it can muster: the category name; the kind of anser being sought; the time, place, and gender hinted at in the clue; and so on.  And when it fees ‘sure’ enough, it decides to buzz.  This is all an instant, intuitive process for a human Jeopardy player, but I felt convinced that under the hood my brain was doing more or less the same thing.”

But does Watson really think?  John Searle answered no, the day after Watson won the Jeopardy competition.  “IBM invented an ingenious program—not a computer that can think,” he declared.  “Watson did not understand the questions, nor its answers, nor that some of its answers were right and some wrong, nor that it was playing a game, nor that it won—because it doesn’t understand anything.”

Some computer scientists have responded to this question of whether a machine can think by asking, “Can a submarine swim?”  Submarines don’t swim the way fish swim or the way some reptiles and mammals swim.  But in some ways, submarines swim better than fish, reptiles, and mammals.  Similarly, Watson certainly doesn’t think the way human beings or other animals think, but it can solve problems and answer difficult questions about the world, in ways that have persuaded many people that is really is thinking.

But can we trust our perception that a machine is thinking?  Alan Turing's "imitation game" assumes that if a machine could successfully imitate a human thinker, so that we could not distinguish between the machine and humans through carrying on an exchange of questions and answers with them, that would show that the machine had achieved something like human-level intelligence.  IBM is hoping to show in a few years that Watson can pass the Turing Test.  Searle has objected, however, that this is not truly a test of human-level intelligence.

The scientists at IBM who built Watson admit that it does not have one crucial feature of human thinking—emotion or feeling.  It did not feel any fear of failure when it played Jeopardy  And it did not feel any pride in winning the game.  The scientists behind Watson did feel such emotions.

When the IBM scientists were testing Watson, they set up Jeopardy games where  Watson was playing against IBM employees who were good Jeopardy players.  When a comedian hired to host practice matches ridiculed Watson’s more obtuse answers (Rembrandt rather than Pollock for a “late ’40s artist”), David Ferrucci, director of the Watson program, complained: “He’s making fun of a defenseless computer.”  When Ferruci brought his daughters to see one of the practice sessions, one of the girls asked: “Daddy, why is that man being so mean to Watson?”

Does human-level intelligence require not just abstract reason but also emotional drives, because human minds care about what they’re thinking and doing?  How could emotion be put into a machine?

One possibility is that an artificial brain might have to be put into an artificial body that would have something like a neuroendocrine system that would generate emotional experience.

Another possibility is building cyborgs—cybernetic organisms—in which human brains and bodies have an interface with intelligent machines.  Thus, human intelligence is augmented by machines, but it’s combined with all the normal emotional drives of human beings.  In a way, many human beings today have already become cyborgs because the intelligence of their brains is augmented by machines through interfaces with computers and smart phones.  Over the next few years, that brain-machine interface will be put inside the human brain and body through neural implants.

Right now, the intelligence of many of us has been augmented by our computers and smart phones. We converse with our machines, and this conversation occurs through brain-machine interfaces in our typing fingers, our speaking voices, our hearing ears, and our seeing eyes.  As these interfaces move to the surface of our bodies (as in Google Glass), electronic skin implants, and then inside our brain, we will have ever more direct access to all of human knowledge.  Google Earth will give us instant views of every place on Earth.  GPS will insure that we are never lost.  Google Books will allow us to download every book that has ever been published.  When we run out of storage space in our heads, we can store our knowledge in Google cloud computing.

This must be what Google cofounder Larry Page had in mind when he said:

"People always make the assumption that we're done with search.  That's very far from the case.  We're probably only 5 percent of the way there.  We want to create the ultimate search engine that can understand anything . . . some people could call that artificial intelligence. . . . The ultimate search engine would understand everything in the world.  It would understand everything that you asked it and give you back the exact right thing instantly."

Page has said that the ultimate goal is for us to merely think of a question, and then we instantly hear or see the answer.

This understanding of everything in the world that cyborgs could have must include understanding emotion.  As I indicated in my post on Morris Hoffman's The Punisher's Brain, Judge Hoffman thinks that most trial judges show "our evolved retributive feelings" when they punish.  "We get a gut, retributive, feeling about the sentence, and then move in one direction or another off that gut feeling based on information about the criminal that affects our views about special deterrence--the likelihood he will reoffend and the crimes he is likely to commit."  So if IBM wants to teach Watson how to be a good judge, they might have to find a way to instill the "gut feelings" that are part of our evolved human nature.

We must wonder about the wisdom of our moving under the rule of "our new computer overlords."  This has already begun.  Most of the buy-sell decisions on Wall Street are being made by computers acting autonomously.  Most of the infrastructure network of North America (electricity, water, and transportation) is controlled by computer systems connected to the Internet.  Doctors are adopting expert computer systems for diagnosing their patients.  The scientists at IBM are improving Watson so that it can make decisions for us in many areas of life.  Much of the research on robot intelligence is funded by DARPA (The Defense Advanced Research Projects Agency), which is aimed at creating autonomous robotic weapons.  The United States military already relies on many weaponized robots.

In his survey of the latest research in AI directed to producing AGI (artificial general intelligence) and then ASI (artificial super-intelligence), James Barrat (Our Final Invention) concludes that there's no reason that ASI will care about human beings, that such super-intelligence will be incomprehensible to us, and that this will lead to the extinction of our species.  He also indicates, however, that only a few AI researchers (like Stephen Omohundro and Eliezer Yudkowsky) share his pessimistic vision of the perils of ASI.  Most of the leading proponents of advanced AI research (like Ray Kurzweil and Rodney Brooks) are optimistic in their utopian vision of ASI as allowing human beings to finally fulfill the human dream, expressed by early modern philosophers and scientists like Descartes and Bacon, of completely mastering nature for human benefit, even including human immortality.

The Turing Test for Emerging Consciousness in a Chinese Room or a Human Brain

I have argued for explaining the human mind as an emergent property of the human brain once it passed over a critical threshold of size and complexity in the evolution of the primate brain.  If that is true, then one might wonder whether technological evolution could do for robots what biological evolution has done for humans.  Is it possible that once computer technology passes over a critical threshold of complexity, comparable to the complexity of the human brain, could a mechanical brain equal or even surpass the intelligence of human beings? 

And if that is possible, what moral, legal, and political questions would this raise?  Must we soon be ruled by robots who are smarter than us?  Or will we use this technology of artificial intelligence to extend our human intelligence, so that we will be as super-intelligent as our machines?  Will our super-intelligent robots demand to be treated as persons with rights?  Will they have a morality like ours?  Or will they be moved by a will to power that is beyond human good and evil?

We can anticipate that such questions about advances in artificial intelligence will become the deepest political questions of the twenty-first century.

I have been thinking about this in my course this semester--"Biopolitics and Human Nature"--because the last part of the course is on the debate over artificial intelligence.  We're reading Alan Turing ("Computing Machinery and Intelligence," Mind, 1950), John Searle ("What Your Computer Can't Know," New York Review of Books, October 9, 2014), Ray Kurzweil (The Singularity is Near, 2005), and James Barrat (Our Final Invention, 2013).

Many years ago, when I first began thinking about this, I was persuaded by Searle's famous Chinese Room argument against the Turing Test for human-level intelligence in a machine.  But now, I think Kurzweil is right in arguing that Searle's Chinese Room doesn't refute the Turing Test.

The Turing Test is the common name today for what Turing originally called the Imitation Game.  He proposed this as the best test of whether a digital computer has achieved intelligence comparable to human intelligence.  (Actually, Descartes proposed a similar test for machine intelligence in his Discourse on Method.)  Put a computer and a human being in separate rooms.  Ask a human being to try to detect which one is the computer by asking questions typed onto pieces of paper slipped under the doors of the rooms.  The computer and human being will answer the questions on pieces of paper, with the computer pretending to be a human being, and the human being trying to show that he is the human being.  If the computer has the intelligence for communicating in language in ways that a good human speaker of the language would interpret as showing human intelligence, then the computer has passed the test.  Writing in 1950, Turing thought that digital computers would begin to pass the test by the year 2000.

In his article, Turing anticipated all of the major objections to his reasoning that have been developed over the past decades.  One of those objections was the argument from consciousness.  He quotes from a Professor Jefferson:  "Not until a machine can write a sonnet or compose a concerto because of thoughts and emotions felt, and not be the chance fall of symbols, could we agree that machine equals brain--that is, not only write it but know that it had written it.  No mechanism could feel (and not merely artificially signal, an easy contrivance) pleasure at its successes, grief when its valves fuse, be warmed by flattery, be made miserable by its mistakes, be charmed by sex, be angry or depressed when it cannot get what it wants" (445-46).

Turing responds to this argument by suggesting that it is unreasonable, because it would throw us into a solipsism that none of us would accept: "This argument appears to be a denial of the validity of our test.  According to the most extreme form of this view, the only way by which one could be sure that a machine thinks is to be the machine and to feel oneself thinking.  On could then describe these feelings to the world, but of course no one would be justified in taking any notice.  Likewise, according to this view, the only way to know that a man thinks is to be that particular man.  It is in fact the solipsist point of view.  It may be the most logical view to hold, but it makes communication of ideas difficult.  A is liable to believe 'A thinks, but B does not,' whilst B believes 'B thinks, but A does not.'  Instead of arguing continually over this point, it is usual to have the polite convention that everyone thinks" (446).

Consciousness is mysterious, Turing observes, because while we all have direct subjective access to our own thoughts and feelings, we have no direct access to the conscious subjective experiences of anyone else.  We can only indirectly infer the consciousness of other people (or of animals) from their behavior.  We must do the same in inferring the conscious thinking of machines.  So Turing's test for the intelligence of digital computers is essentially the same test that we all employ in our lives everyday to infer the conscious thoughts and feelings of other human beings.

Searle's Chinese Room argument is his way of stating the argument from consciousness as refuting the Turing Test.  And Kurzweil's response to Searle is a restatement of Turing's response to Jefferson's objection.

Searle insists that his Chinese Room Argument shows that a computer could pass the Turing Test without having any conscious understanding of anything.  He explains: "Imagine someone who doesn't know Chinese--me, for example--following a computer program for answering questions in Chinese.  We can suppose that I pass the Turing Test because, following the program, I give the correct answers to the questions in Chinese, but all the same, I do not understand a word of Chinese.  And if I do not understand Chinese on the basis of implementing the computer program, neither does any other digital computer solely on that basis."

According to Searle, this shows that a computer programmed for communicating in language has a syntax but no semantics.  The computer can manipulate linguistic symbols according to rules of syntax, but it has no semantic understanding of the meaning of what it says, because it has no conscious experience of anything, no subjective thoughts or feelings.

Kurzweil responds by arguing that Searle's Chinese Room won't work, because a computer could not perfectly simulate understanding Chinese--pass a Chinese Turing Test--if it did not really understand Chinese.  After all, for a human being to persuade us that he understands a language, he must actually understand it. 

Human brains can understand language because human brains are amazingly complex.  Any computer that could understand language would have to be as complex as the human brain.  So far, no computer has ever reached that level of complexity.  But, in principle, this is possible.  Once we understand the complexity of the human brain, and once a computer has replicated the complexity of the human brain, Kurzweil argues, we will recognize this breakthrough when a computer can persuade us, through language and other intelligent behavior, that it has conscious thoughts and feelings comparable to human beings.

Oddly enough, Searle actually concedes this possibility of creating an "artificial brain."  Searle writes:  "An artificial brain has to literally create consciousness, unlike the computer model of the brain, which only creates a simulation.  So an actual artificial brain, like the artificial heart, would have to duplicate and not just simulate the real causal powers of the original.  In the case of the heart, we found that you do not need muscle tissue to duplicate the causal powers.  We do not know enough about the operation of the brain to know how much of the specific biochemistry is essential for duplicating the causal powers of the original.  Perhaps we can make artificial brains using completely different physical substances as we did with the heart.  The point, however, is that whatever the substance is, it has to duplicate and not just simulate, emulate, or model the real causal powers of the original organ.  The organ, remember, is a biological mechanism like any other, and it functions on specific causal principles."

Kurzweil agrees with this: human-level artificial intelligence will have to arise in an artificial mechanical brain that copies the organization and causal powers of the human brain.  The evolution of the human brain from the primate brain shows that as the primate brain expanded in size and complexity, it eventually passed over a critical threshold in which new patterns emerged that gave rise to human conscious intelligence.

Like Turing, Kurzweil admits that consciousness is mysterious, because it somehow emerges from the brain, but unlike the brain, consciousness is not objectively observable or measurable, because the subjective experience of consciousness can be directly known only in our personal mental experience.  Deciding whether entities outside of us are conscious depends on an indirect inference from the behavior of those entities, and thus we cannot prove that entities outside of us are conscious through objective tests.

This explains why the scientific study of consciousness is so difficult.  Indeed, Kurzweil concludes that the question of consciousness is ultimately not a scientific question at all, because it's a question that cannot be answered finally through objectively measurable tests.

But despite this mystery of consciousness, we can scientifically observe the evolutionary history of the brain and the evolutionary emergence of the human mind in the brain.  We are still left wondering, however, whether the next step in evolution is the technological evolution of human-level intelligence in an artificial brain.

Darwinian Natural Right in the Punisher's Brain

"Evolution built us to punish cheaters."

Thus does Morris Hoffman begin his fascinating new book--The Punisher's Brain: The Evolution of Judge and Jury (Cambridge University Press, 2014).  Hoffman is a trial judge in Denver, Colorado, and the most engaging feature of this book is that he speaks as a trial judge who wants to understand his experience in deciding whether and how to punish people charged with law-breaking.  The students in my course on "Biopolitics and Human Nature" this semester have enjoyed how he illustrates his points with stories about the cases he has had.

Hoffman first developed his interest in law and biology from attending the conferences of the Gruter Institute for Law and Behavioral Research organized first by Margaret Gruter and later by her granddaughter, Monika Gruter Cheney.  He was then introduced to law and neuroscience by joining the John D. and Catherine T. MacArthur Foundation's Research Network on Law and Neuroscience, directed by Owen Jones.  From these two groups, he learned how to apply evolutionary psychology and behavioral neuroscience to the study of law.

I largely agree with Hoffman, because most of what he says I see as the application of Darwinian natural right to the study of law.  My only disagreement is that in relying on neuroscience, and especially brain-scanning experiments, he does not acknowledge, much less respond to, the many criticisms of what Sally Satel and Scott Lilienfeld have called "mindless neuroscience"--the exaggerated claims for brain-scanning as mind-reading that ignore the problems in inferring the thoughts and feelings of the mind from neural correlates in the brain.

I have written posts on the "brain-imaging fallacy" here and here.  The fundamental problem is the mystery of consciousness--that our only direct access to conscious thoughts and feelings is through our own internal subjectivity, and that any inference of what's happening in the mind from what is happening in the brain must always be uncertain and imprecise.  Hoffman could have acknowledged such problems in neuroscience and brain-scanning without weakening his general argument, which is supported by many different lines of reasoning.

Hoffman begins with a story about a murder trial.

"Several years ago, I presided over a first-degree murder trial in which a young Czech émigré was charged with stabbing his Brazilian au pair girlfriend.  The crime took place in the au pair's bedroom, in the basement of her employer-family's house.  The young man stabbed her seventy-four times.  He confessed to the murder but denied it was premeditated.  Despite his denial, the premeditation evidence was pretty strong.  He not only entered the bedroom through a window, armed with a knife and carrying some duct tape, but he also admitted to police that a few days before the killing he tried to dig a small grave in a remote field but gave up because the ground was frozen."

"On the other hand, he testified that he regularly went through the window for late-night visits with her, and that he went there that night not to kill her but only to see if she would change her mind about breaking up.  he claimed he had the knife and duct tape because he was moving.  As for the grave, he testified that he started to dig the hole in the field to 'bury her memory,' and that all he intended to bury there were a few items of personal property that reminded him of her.  When he went to see her that final time, and she told him she was set on leaving him, he 'snapped.'"

"But he didn't say the word 'snapped.'  What he said was, 'A darkness came across my eyes.'  He even said it a second time in cross-examination.  It seemed oddly and rather beautifully phrased, and vaguely familiar.  Neither of the lawyers asked him about it.  Long after the jury convicted him of first-degree murder, and I sentenced him to the mandatory life in prison without the possibility of parole, it hit me.  'Darkness covered his eyes,' and variations of that phrase, are used over and over by Homer to describe many of the battle deaths in The Iliad." (1-2)

Judge Hoffman offers this as an illustration of what he has seen as a common occurrence in criminal law cases, where there is no dispute over the fact that the defendant committed the crime, but the question for judges and juries is what was going through the defendant's mind at the time of the crime.  Was the defendant acting purposefully, knowingly, recklessly, or negligently in committing the crime?  In this case, the difference was between a purposeful, first-degree murder punished with a life sentence in prison and a knowing, second-degree murder punished in Colorado with a mandatory prison sentence of 16-48 years.

In such cases, modern juries and judges must decide blameworthiness and punishment through judging the harm and the intentionality of wrongdoers' conduct.  The greater the harm and the clearer the intentionality of the conduct, the greater the blame and punishment that it will elicit.

In doing this, Hoffman argues, modern juries and judges are doing what our evolutionary ancestors have been doing for over 100,000 years in deciding how to respond to wrongdoers.  And while much of the law is a cultural construction that reflects the historical contingencies that have shaped each legal system, there is also a universal pattern in law that manifests evolved human nature.

Human beings have always faced what Hoffman calls the Social Problem, which is similar to what others have called the Collective Action Problem, the Commitment Problem, the Trust Problem, or the Altruism Problem.  The problem arises from our human nature as both selfish and social animals, so that we must always face the question: cheat or cooperate?  We are inclined to cheat others in our group whenever cheating would be to our selfish advantage.  But we are also inclined to cooperate, because living in cooperative groups has always given us long-term advantages in the struggles of life.  We have evolved instincts both to cheat and to cooperate.  But we also have a third evolved instinct--to punish cheaters in order to reduce cheating and increase cooperation by increasing the costs of cheating.

Hoffman explains our punishment of cheaters as moving through three levels.  Through first-party punishment, we punish ourselves with conscience and guilt.  Through second-party punishment, we punish our tormentors with retaliation and revenge.  Through third-party punishment, we act as a group in punishing wrongdoers with retribution.  Judges and jurors are acting as third-party punishers.  Hoffman's argument is that the human brain has been shaped by biological evolution to have the instinctive propensities for punishment at all three levels.

Moreover, he argues, at all three levels, we are guided by three rules of right and wrong rooted in our evolved human nature to secure property and promises.  Rule 1: Transfers of property must be voluntary.  Rule 2:  Promises must be kept.  Rule 3:  Serious violations of Rules 1 and 2 must be punished.

Hoffman interprets "property" in a broad sense as starting with self-ownership and encompassing one's life, health, and possessions, as well as the life, health, and possessions of one's family and others to whom one is attached.  (Although he does not mention John Locke, Hoffman here echoes Locke's argument for self-ownership as the ground of property rights.  Indeed, it seems to me that Hoffman's whole argument for the evolution of punishment supports Locke's account of how the instinctive propensities for punishment sustain social order.)  Understood in this broad way, Rule 1 embraces criminal law and tort law, while Rule 2 embraces contract law.

Classical liberals or libertarians could embrace this as a good statement of their claim that the primary purpose of law is to punish force and fraud and secure the liberty of individuals to live as they please so long as they do not harm others.

Hoffman supports his argument for these kinds of rules and punishment being rooted in evolved human nature with at least ten kinds of evidence. 

(1)  Economic game experiments (such as the Ultimatum Game, the Public Goods Game, and the Trust Game) can show, both within our culture and cross-culturally, that most human beings are inclined to cheat, to cooperate, and to punish cheaters at all three levels. 

(2)  Comparison with other species of animals can show that some other animals show similar behavioral inclinations. 

(3)  We can see how hormones (such as oxytocin and testosterone) support these inclinations. 

(4)  We can study the brain as a behavioral fossil record of evolution. 

(5)  We can look to anthropology for evidence that these instinctive inclinations are human universals. 

(6)  We can also look to anthropology for evidence of the law in primitive societies that might show instinctive behavior like that of our distant evolutionary ancestors. 

(7)  We can study experimental surveys in which people are presented with hypothetical legal scenarios, and they are asked to judge blameworthiness and punishment, which allow us to see if they show these instinctive inclinations. 

(8)  If people are in brain-scanning machines (fMRI), we can conduct surveys or have them play economic games, and then we can try to infer the neural correlates of the feelings and thoughts that drive our instincts for punishing. 

(9)  We can look at the history of law to see patterns of punishing that manifest our evolved instincts. 

(10)  Finally, we can look at young human infants for evidence of those instincts arising early in life.

Every one of these lines of evidence is rightly open to dispute.  But, at least, this wide range of evidence shows that Hoffman's biolegal theory cannot be dismissed as a "just-so story" that is untestable.

FIRST-PARTY PUNISHMENT: CONSCIENCE AND GUILT
Most human beings punish themselves for cheating through conscience and guilt.  Guilt is retroactive blame, feeling pained by the thought of our past misconduct.  Conscience is prospective blame, imagining the pain we would feel if we were to engage in some misconduct.  Such conscience and guilt requires empathy--being able to imaginatively put ourselves in the situation of others and feel the pain they might feel from our injuring them.

Hoffman points to the evidence for the neural correlates of conscience and guilt in particular parts of the brain, and for the diminished capacity for conscience and guilt when there is some innate or acquired abnormality in these parts of the brain.  So, for example, reduced connectivity in the ventromedial prefrontal cortex (vmPFC) of the brain seems to be associated with psychopathic psychology.  Psychopaths--those with little or no capacity for conscience and guilt--are the exception that proves the rule that most human beings have some instinctive propensity to punish themselves for violating moral rules against harming others.

SECOND-PARTY PUNISHMENT:  RETALIATION AND REVENGE
If conscience and guilt fail to restrain us from cheating, we must then worry about the punishment coming from our victims or their family and friends.  For most of our evolutionary history, the primary punishment of wrongdoers was retaliation and revenge (delayed retaliation). 

The law of self-defense--that everyone has the right to retaliate against attacks on their lives, their health, or their property--is universal, and it is supported by neural circuitry in the amygdala, the insula, the vmPFC, the cingulate, and the dorsolateral prefrontal cortex (dlPFC).  If people play the Ultimatum Game while they're in a brain-scanning machine, we can see the enhanced activity of this neural circuitry when they refuse unfair offers, and thus inflict a costly punishment on the other player.

And yet this propensity for retaliation and revenge is dangerous, because it can easily become too extreme.  Too much punishment can be as disruptive to social order as too little punishment.  This is what turns a state of nature from a state of peace to a state of war.  To avoid this, we need the rule of law or third-party punishment, because third-party punishers tend to be more dispassionate in their retaliation.

THIRD-PARTY PUNISHMENT: RETRIBUTION
The emotions that we feel when we punish wrongdoers for harming others can be very strong, but usually they are not as strong as they are when we are punishing those who have harmed us directly.  For that reason, third-party punishers can move towards a more impartial judgment, which is what we look for in the rule of law.

Treating our families as extensions of ourselves turns second-party punishment into third-party punishment.  But this familial third-party punishment is not likely to be as impartial as punishment coming from someone who is unrelated to the victim or the wrongdoer.  Originally, those dominant individuals who acted as mediators or judges of disputes in the band or tribe exercised third-party punishment on their own.  But in some special cases, they might have delegated this to select groups of people, which would have acted as the first juries.

From brain-scanning, there is some evidence for the neural correlates of third-party punishment.  The right dorsolateral prefrontal cortex seems to be active both when people are engaged in third-party punishment (in weighing punishment for hypothetical criminal behavior) and when they are engaged in second-party punishment (in retaliating against unfair players in economic games).  This suggests that the modern legal system--with a centralized government enforcing punishment--could have been built on the cognitive mechanisms that evolved for retaliation and revenge.

The detailed rules and procedures for third-party punishment in modern legal systems show the vagaries of historical contingency in the cultural evolution of law that is highly variable across legal systems.  But still these modern rules and procedures can manifest general patterns rooted in ancient human instincts for punishing.  For example, one ancient form of punishment for the most severe crimes was ostracism or banishment from the community.  Although prisons are a relatively new invention in legal history, Hoffman observes, imprisonment can be seen as a new way to punish people by ostracizing or banishing them from the community, either temporarily or permanently.

THE GUT FEELINGS OF LEGAL PUNISHMENT
Judge Hoffman thinks that most trial judges show "our evolved retributive feelings" when they punish.  "We get a gut, retributive, feeling about the sentence, and then move in one direction or another off that gut feeling based on information about the criminal that affects our views about special deterrence--the likelihood he will reoffend and the crimes he is likely to commit" (345). 

Surely, many readers will be disturbed to learn that most trial judges are guided in their judgments by "gut feelings."  After all, doesn't Judge Hoffman indicate that third-party punishing should be "more dispassionate" and impartial than second-party punishing (138)?  Or is he saying that even if they are "more dispassionate," judges cannot, and should not, be completely free from the moral passions that instinctively drive legal punishment?

I agree with Judge Hoffman about the importance of evolved "gut feelings" for law.  I see this as very similar to what I say in Darwinian Natural Right (61-83) about "natural morality."  The moral judgments expressed in law, like all moral judgments, require a combination of moral emotion and moral reason as evolved moral instincts.  Reason can elicit, direct, and organize feelings.  But pure reason alone could never create moral right or wrong, because it cannot create moral feelings.

We do not commit a naturalistic fallacy when we move from natural facts to moral values--or from is to ought--if we limit ourselves to the claim that for the kind of species that we are, certain feelings are predictably aroused by certain facts, and the experience of those feelings is the only ground for making moral judgments.  Except for psychopaths, most human beings feel the moral emotions of conscience, guilt, retaliation, revenge, and retribution in response to the facts of criminal misconduct.

Judge Hoffman shows this combination of evaluative emotion and factual reasoning in his book.  The gut feelings of trial judges are rational if they are "based on information about the criminal that affects our views."  In telling the stories of his trials, he always relates the facts of the case with the expectation that these facts will elicit the same "gut feelings" that they elicited in the judges and the jurors as they reached their decision.

So, for example, when Judge Hoffman relates the story of the young Czech murder, he presents the evidence that he and the jury saw as evidence that the man was lying about his murder being unpremeditated.  He assumes that his readers will agree with this factual reasoning, and that they will also feel the retributive emotions that demanded a first-degree murder conviction.

Thus, the gut feelings of Judge Hoffman and his jurors do not confirm to any cosmically objective standard of right and wrong, but neither are they expressing purely arbitrary personal emotions.  As Judge Hoffman indicates, the instinctive propensity to punish cheaters to enforce cooperation has been evolutionarily adaptive for the human species, but not necessarily for other species.  "Social cooperation is not an abstract good" (26).  That is to say that evolution does not enforce some cosmic good like Kant's Categorical Imperative.  Rather, what is good is relative to each species.

But that species-specific good for human beings does have an intersubjective objectivity, in that the facts of each case rightly understood should evoke similar moral emotions in most normal human beings.  We can agree that legal doctrines and legal decisions are just as long as they conform to our evolved instinctive feelings.

By contrast, those legal doctrines and decisions that violate our instinctive feelings can be judged to be unjust.  Judge Hoffman indicates this in his discussion of "legal dissonances."  He notes that juries can nullify laws they find offensive by refusing to convict defendants who have clearly violated those laws.  For example, jurors often acquit defendants charged with buying small amounts of illegal drugs, because the jurors don't see this as a serious crime.  Hoffman then defines "legal dissonance" as "a narrow segment of nullification involving legal rules that seem to conflict with our evolved intuitions, especially our evolved notions of blameworthiness" (252).

A good example of a legal doctrine that conflicts with our evolved intuitions is the felony murder rule, which says that if anyone dies in the commission of a felony, the felon is guilty of first-degree murder, even if the felon did not intend to cause the death.  Hoffman tells the story of a famous case in Colorado:

"In November of 1997, a nineteen-year-old woman named Lisl Auman left her abusive boyfriend, and was staying overnight at a girlfriend's apartment.  The girlfriend was having her own boyfriend problems, and she asked a couple friends of hers, including a skinhead named Matthaeus Jaehnig, to act as muscle when the two women went to retrieve her belongings from their respective exes.  Auman's ex was not at home, so one of the other men cut the lock off the apartment door, as Jaehnig stayed outside as lookout.  Auman went in and took her belongings, plus several things belonging to her ex.  Another resident of the apartment complex became suspicious, wrote down Jaehnig's license number, and called police.  A high-speed chase ensued, during which Auman told Jaaehnig several times that she was afraid, and he should stop the car.  Instead, Jaehnig drove the girlfriend's apartment, and even shot at police a few times along the way.  Before police managed to catch up to them, Jaehnig and the others split up.  Police arrested Auman, handcuffed her, and placed her in the back of a police car.  They even drove the squad car a little further away in the parking lot to get away from the scene.  Other officers pursued Jaehnig.  by the time the five-minute foot chase was over, Jaehnig had shot and killed one of the officers, and then turned the gun on himself.  At the moment of the officer's and Jaenig's deaths, Auman was cuffed and sitting in the back of the police car hundreds of feet away." (254)

Under Colorado's version of the felony murder rule, Auman was charged and convicted of first-degree murder, and sentenced to life in prison without parole.  When this case was appealed to the Colorado Supreme Court, the Court upheld this application of the felony murder rule, but the Court reversed the conviction, because it detected an error in the jury instruction on the burglary charge.  But since this error had nothing to do with the outcome of the case, this did not justify the reversal.  Clearly, Judge Hoffman observes, the Court used this as "an excuse to undue a profoundly unjust result" (255).

Hoffman points out that the felony murder rule was originally adopted by a few American states through an "academic accident" (302).  It was imported from England because some English legal commentators incorrectly reported that it was widely adopted in England.  In fact, it had been adopted in only a few cases in England in the 1880s; it was criticized by many commentators; and it was abolished by Parliament in 1957.

Hoffman thinks there are good reasons to abolish the felony murder rule in the few states were it exists.  "It conflicts with our deepest notion that we blame only intentional wrongs, it is not itself deeply-grounded in our evolution or in our jurisprudence, it grossly over-punishes in the eyes of ordinary citizens, and in its one hundred thirty years of existence, it has already winked out either by outright abolition or by exceptions that swallow it" (303-304).

Judge Hoffman never uses terms like "natural law" or "natural right."  But isn't this a clear case of natural-law reasoning, in which we see that some laws are unjust if they violate our evolved moral instincts?

Some posts on related topics can be found here, here, and here.