Faculty Bibliography

Making appropriate choices often requires the ability to learn the value of available options from experience. Parkinson's disease is characterized by a loss of dopamine neurons in the substantia nigra, neurons hypothesized to play a role in reinforcement learning. Although previous studies have shown that Parkinson's patients are impaired in tasks involving learning from feedback, they have not directly tested the widely held hypothesis that dopamine neuron activity specifically encodes the reward prediction error signal used in reinforcement learning models. To test a key prediction of this hypothesis, we fit choice behavior from a dynamic foraging task with reinforcement learning models and show that treatment with dopaminergic drugs alters choice behavior in a manner consistent with the theory. More specifically, we found that dopaminergic drugs selectively modulate learning from positive outcomes. We observed no effect of dopaminergic drugs on learning from negative outcomes. We also found a novel dopamine-dependent effect on decision making that is not accounted for by reinforcement learning models: perseveration in choice, independent of reward history, increases with Parkinson's disease and decreases with dopamine therapy.

We review and synthesize recent neurophysiological studies of decision making in humans and nonhuman primates. From these studies, the basic outline of the neurobiological mechanism for primate choice is beginning to emerge. The identified mechanism is now known to include a multicomponent valuation stage, implemented in ventromedial prefrontal cortex and associated parts of striatum, and a choice stage, implemented in lateral prefrontal and parietal areas. Neurobiological studies of decision making are beginning to enhance our understanding of economic and social behavior as well as our understanding of significant health disorders where people's behavior plays a key role.

We present an overview of recent paradigms used for studying visual information and reward processing in the human and monkey oculomotor pathways. Current evidence indicates that eye movements made during visual search tasks rely on neural computations similar to those employed when eye movements are planned and executed to obtain explicit rewards. These data suggest that human eye movements originate from the processing of (predominantly visual) sensory information, feedback about previous errors, and expectations about factors, such as reward. We conclude that these properties make the saccadic system an ideal model for studying both the behavioral and neural mechanisms for human voluntary and involuntary choice behavior.

Over the course of the past decade, neurobiologists have become increasingly interested in concepts and models imported from economics. Terms such as "risk," "risk aversion," and "utility" have become commonplace in the neuroscientific literature as single-unit physiologists and human cognitive neuroscientists search for the biological correlates of economic theories of value and choice. Among neuroscientists, an incomplete understanding of these concepts has, however, led to a growing confusion that threatens to check the rapid advances in this area. Adding to the confusion, notions of risk have more recently been imported from finance, which employs quite different, although formally related, mathematical tools. Of course, the mixing of economic, financial, and neuroscientific traditions can only be beneficial in the long run, but truly understanding the conceptual machinery of each area is a prerequisite for obtaining that benefit. With that in mind, I present here an overview of economic and financial notions of risk and decision. The article begins with an overview of the classical economic approach to risk, as developed by Bernoulli. It then explains the important differences between the classical tradition and modern neoclassical economic approaches to these same concepts. Finally, I present a very brief overview of the financial tradition and its relation to the economic tradition. For novices, this should provide a reasonable introduction to concepts ranging from "risk aversion" to "risk premiums."

Choosing the most valuable course of action requires knowing the outcomes associated with the available alternatives. The striatum may be important for representing the values of actions. We examined this in monkeys performing an oculomotor choice task. The activity of phasically active neurons (PANs) in the striatum covaried with two classes of information: action-values and chosen-values. Action-value PANs were correlated with value estimates for one of the available actions, and these signals were frequently observed before movement execution. Chosen-value PANs were correlated with the value of the action that had been chosen, and these signals were primarily observed later in the task, immediately before or persistently after movement execution. These populations may serve distinct functions mediated by the striatum: some PANs may participate in choice by encoding the values of the available actions, while other PANs may participate in evaluative updating by encoding the reward value of chosen actions.

The basal ganglia appear to have a central role in reinforcement learning. Previous experiments, focusing on activity preceding movement execution, support the idea that dorsal striatal neurons bias action selection according to the expected values of actions. However, many phasically active striatal neurons respond at a time too late to initiate or select movements. Given the data suggesting a role for the basal ganglia in reinforcement learning, postmovement activity may therefore reflect evaluative processing important for learning the values of actions. To better understand these postmovement neurons, we determined whether individual striatal neurons encode information about saccade direction, whether a reward had been received, or both. We recorded from phasically active neurons in the caudate nucleus while monkeys performed a probabilistically rewarded delayed saccade task. Many neurons exhibited peak responses after saccade execution (77 of 149) that were often tuned for the direction of the preceding saccade (61 of 77). Of those neurons responding during the reward epoch, one subset showed direction tuning for the immediately preceding saccade (43 of 60), whereas another subset responded differentially on rewarded versus unrewarded trials (35 of 60). We found that there was relatively little overlap of these properties in individual neurons. The encoding of action and outcome was performed by largely separate populations of caudate neurons that were active after movement execution. Thus, striatal neurons active primarily after a movement appear to be segregated into two distinct groups that provide complimentary information about the outcomes of actions.