Moral zombies: why algorithms are not moral agents — TEI-tagged view

Carissa VélizpersName (2021date). “Moral zombies: why algorithms are not moral agents.” AI & SOCIETYtitle·j.

Object of study — whether algorithmsterm that produce morally consequential outputs qualify as moral agentsterm·def

Research question — whether moral agencyterm is independent of sentient experience

Contribution — a moral-zombie argument that behavioral impact is insufficient for agency and that accountabilityterm remains with human designers and users

Full text — tagged

Moral zombiesterm: why algorithmsterm are not moral agentsterm·def Abstract In philosophy of mind, zombies are imaginary creatures that are exact physical duplicates of conscious subjects for whom there is no first-personal experience. Zombies are meant to show that physicalism—the theory that the universe is made up entirely out of physical components—is false. In this paper, I apply the zombie thought experiment to the realm of morality to assess whether moral agencyterm is something independent from sentienceterm. Algorithmsterm, I argue, are a kind of functional moral zombieterm, such that thinking about the latter can help us better understand and regulate the former. I contend that the main reason why algorithmsterm can be neither autonomous nor accountable is that they lack sentienceterm. Moral zombiesterm and algorithmsterm are incoherent as moral agentsterm·def because they lack the necessary moral understanding to be morally responsible. To understand what it means to inflict pain on someone, it is necessary to have experiential knowledge of pain. At most, for an algorithmterm that feels nothing, ‘values’ will be items on a list, possibly prioritised in a certain way according to a number that represents weightiness. But entities that do not feel cannot value, and beings that do not value cannot act for moral reasons.

Keywords: Algorithmsterm, Moral agencyterm, Moral responsibilityterm·def, Autonomous systems, Zombies, Accountabilityterm, Autonomyterm, Sentienceterm, Consciousness, Reasons-responsivenessterm Introduction In philosophy of mind, zombies are imaginary creatures designed to illustrate issues related to physicalism and consciousness. Zombies are exact physical duplicates of conscious subjects for whom there is no first-personal experience. In addition to being physically just like us, molecule for molecule, they are also our functional duplicates. Zombies behave indistinguishably from human beingsterm—they complain about the weather, they cry when they watch sad movies, and they have been known to discuss philosophy conundrums for hours at a time. Unlike human beingsterm, they lack conscious experience. It is nothing there is like to be a zombie—they do not suffer from cold when it is cold, they do not feel sadness when they cry, and philosophical puzzles do not worry or excite them.

Zombies are meant to show that physicalism—the theory that the universe is made up entirely out of physical components—is false. If there could be creatures that act like us without conscious experience, then consciousness must be something above and beyond our physical bodies.

The zombie thought experiment is yet to be applied to the realm of morality, but there is an analogous question to be explored regarding sentienceterm and moral agencyterm. Applying the zombie thought experiment to moral agencyterm can help us assess whether moral agencyterm is something independent of conscious experience, or sentienceterm. By sentienceterm I mean the capacity, at a minimum, to have a subjective experience of pleasure and pain. I wish to remain agnostic as to whether subjective experience is synonymous with sentienceterm. For the purposes of this paper, I will focus on sentienceterm because it is conceivable that there could be conscious experience without feelings of pain or pleasure, and without emotions, and I take these latter experiences to be what matter most for moral agencyterm. Delving into these issues, however, is beyond the scope of this paper and unnecessary for my purposes.

In what follows, I first present the case for algorithmsterm being akin to moral zombiesterm, and for the significance of the moral zombieterm thought experiment. I then suggest that much of the literature on algorithmsterm assumes a degree of moral agencyterm in algorithmsterm. In turn, most accounts of moral agencyterm defend either some version of autonomyterm or some version of moral responsibilityterm·def as constitutive of moral agencyterm, or both. I go on to analyse whether algorithmsterm and moral zombiesterm are autonomous in the relevant sense for moral agencyterm, and argue that they are not. I then argue that moral responsibilityterm·def is more relevant to assessing moral agencyterm than autonomyterm, but neither algorithmsterm nor moral zombiesterm are morally responsible either. I contend that the main reason why algorithmsterm can be neither autonomous nor accountable is that they lack sentienceterm. Moral zombiesterm are incoherent as moral agentsterm·def because they lack the necessary moral understanding to be morally responsible. To understand what it means to inflict pain on someone, it is necessary to have experiential knowledge of pain. I end the paper by responding to three possible objections.

Algorithmsterm as moral zombiesterm Moral zombiesterm would be creatures who act indistinguishably from us as moral agentsterm·def, but for whom there is nothing it is like to be them. Moral zombiesterm would act like us without feeling like us. Like human beingsterm, they would be able to do good by donating to charity, protecting vulnerable people against injustice, and respecting rights. They would also have the power to do evil by violating rights and causing unnecessary suffering—they could insult, betray, threaten, and physically aggress others. Unlike human beingsterm, they would not feel pain, pleasure, empathy, intimacy, remorse, guilt, shame, or any other moral emotion. A moral zombieterm would not rejoice at saving a life or suffer guilt from taking one.

Moral zombiesterm, as described, seem to be conceivable creatures, which shows that the power to have a moral impact in the world is independent from sentienceterm. But we already know that from hurricanes and other natural phenomena that can cause harm without there being any agency. The relevant question for ethics is whether moral zombiesterm are conceivable as moral agentsterm·def . If they are not, then the thought experiment would suggest that where there is moral agencyterm, there is sentienceterm.

Moral zombiesterm may not only be a thought experiment. It could be argued that something like moral zombiesterm are already among us; they are often called algorithmsterm, robots, or AI (I will use these terms more or less interchangeably). Automated systems may not look like us (yet), but this detail is arguably morally irrelevant (at least for the question of moral agencyterm). We can imagine robots looking like human beingsterm in the not too distant future. While AI is not yet and will likely never be a physical duplicate of human beingsterm, it can be considered a functional duplicate in some respects—which is why machines can replace human beingsterm in increasingly more tasks. Algorithmsterm already resemble us in some of the decisions they are making, and the moral impact they can have on the world, with people getting hired, sacked, rewarded, and even jailed on account of them. They are not like us, however, in that there is nothing it is like to be them. Thus, thinking about algorithmsterm can help us think through the moral zombieterm thought experiment, and thinking about moral zombiesterm can help us better understand algorithmsterm.

The first objection someone might level at my argument is that we cannot know for sure that algorithmsterm or machines are not sentient. Admittedly, I cannot prove beyond all shadow of a doubt that there is nothing it is like to be an algorithmterm. But, for that matter, I cannot prove that rocks are not sentient either. If panpsychists are right, even rocks may have some degree of consciousness or proto-consciousness. There is, however, not enough evidence to suggest that algorithmsterm (or rocks) have a mind or can feel.

It might be argued that, out of epistemic humility, if algorithmsterm behave as us, we should treat them much like we treat human beingsterm ( Danaher 2020 ; Sparrow 2004 ). At the moment, however, algorithmsterm resemble us only in some of the tasks and functions they perform in society, but not in a more global sense. In that regard, they are not full zombies, but merely something like functional moral zombiesterm. Algorithmsterm today can select job candidates or assess the probability of someone committing a crime, but they cannot develop a passion for French comedies or be moved by kindness.

There may come a time when machines are so similar to us that caution may be warranted ( VélizpersName 2016 ). It is likely that they would need to have bodies similar to ours. There might not be such a thing as a disembodied mind ( Varela et al. 1991 ). And it might be that only beings with biological bodies can be sentient. Part of what makes us feel and think the way we do is our heart beating faster when we are excited, our blood pressure dropping when we experience sadness, sweat running down our temples when we are scared, our skin curling into goose bumps when we are awestruck. The burden of proof, then, seems to be on whoever wants to argue that algorithmsterm are sentient.

Moral agencyterm and algorithmsterm There is a tendency in the literature to describe automated systems in ways that seem to imply some degree of moral agencyterm 1 ( Sharkey 2017 ). In some cases, implicit references to moral agencyterm seem to be a mere rhetorical device. In the spirit of Daniel Dennett’s intentional stance, sometimes it can be useful to describe artefacts as if they had intentions and beliefs, even when we think them mindless. Wendell Wallach and Colin Allen write, for example, that ‘Driverless systems put machines in the position of making split-second decisions that could have life or death implications’ (2009: 14). In the same vein, Robert Sparrow writes that what makes weapons systems automated is ‘that they have the capacity to choose their own targets’ (2007: 70). Even though Sparrow goes on to argue that autonomous weapons systems are not morally responsible, the language of moral decision-making seems appropriate only in reference to moral agentsterm·def. Referring to autonomous systems in ways that are accurate only when describing moral agentsterm·def is morally significant. As Noel Sharkey puts it: ‘[Some terms] act as linguistic Trojan horses that smuggle in a rich interconnected web of human concepts that are not part of a computer system or how it operates. Once the reader has accepted a seemingly innocent Trojan term (…) it opens the gate to other meanings associated with the natural language use of the term’ ( Sharkey 2012 : 793).

Perhaps, this tendency to talk of autonomous systems as if they were moral agentsterm·def, or the messy debate on moral agencyterm and AI, has led some writers to suggest that we should stop asking ourselves if autonomous systems are truly moral agentsterm·def ( Behdadi and Munthe 2020 ). 2 But that approach seems unsatisfying. The reason we ask ourselves if AIs are or can be moral agentsterm·def is not out of metaphysical curiosity, but rather because we care about the practical implications. If AIs are not moral agentsterm·def, then someone needs to take responsibility for what they do. If AIs are moral agentsterm·def, then presumably we should be treating them accordingly (suing them when they break the law, sending them to jail, and considering them bearers of rights, among other practices).

Although most writers only use the language of moral agencyterm metaphorically or pragmatically, in some cases, there does seem to be an explicit belief that autonomous agentsterm·def can be moral agentsterm·def. For instance, when writing about self-driving cars, Mark Coeckelbergh argues that ‘all agency is entirely transferred to the [car]’ (2016: 754).

Of course, different authors mean different things when referring to moral agencyterm. A dominant way of categorising moral agencyterm in the AI literature is that of James H. Moor ( Moor 2009 ). He proposes four categories of ethical agency: (1) ethical impact agentsterm·def (agentsterm·def whose actions have ethical consequences; most or all autonomous systems count as ethical impact agentsterm·def); (2) implicit ethical agentsterm·def (those that have been designed with ethics in mind, e.g. ATMs); (3) explicit ethical agentsterm·def (agentsterm·def that can act from ethics, and not merely in accordance with ethics; they can identify ethical problems and work out solutions for themselves); (4) full ethical agentsterm·def (they can do what explicit ethical agentsterm·def do and, in addition, they have consciousness, intentionality, and free will). My claim in this paper is that explicit ethical agentsterm·def and full ethical agentsterm·def are one and the same category. In other words, I claim we will not get an agentterm·def that can identify ethical problems and respond adequately to them without sentienceterm.

Given that my argument pivots on sentienceterm, the most interesting perspective (and contrary to mine) is that of Luciano Floridi and J.W. Sanders, who not only argue that artificial autonomous agentsterm·def can be considered moral agentsterm·def, but also that they constitute an instance of ‘mind-less morality’ (2004, 351). In disagreement with Floridi and Sanders, I contend that sentienceterm is necessary for moral agencyterm, because conscious experience of the kind that allows for feelings like pleasure, pain, and empathy seems necessary for moral agencyterm.

An agentterm·def is, roughly, an entity that can be the source of action. Most accounts of moral agencyterm in philosophy defend either some version of autonomyterm or some version of moral responsibilityterm·def as constitutive of moral agencyterm, or both. The relationship between these concepts is fraught with controversy. Some philosophers think that whenever there is one, the other follows, while others argue that autonomyterm and moral responsibilityterm·def are independent from each other. For our purposes, it is not essential to establish the exact relationship between autonomyterm and moral responsibilityterm·def. What matters is that, under any plausible account of autonomyterm that is relevant to morality, and under any plausible account of moral responsibilityterm·def, neither moral zombiesterm nor algorithmsterm can be moral agentsterm·def, or so I argue.

Autonomyterm and moral agencyterm Nomy Arpaly argues that there are at least eight ways in which ‘autonomyterm’ has been conceptualised in the literature: as personal efficacy, psychological independence, having a moral right to self-determination, authenticity, having a coherent self-image, being heroic, self-governanceterm·var, and being responsive to reasons. In what follows I go through each of these and argue that many of these accounts of autonomyterm do not seem relevant to morality, and the ones that do (self-governanceterm·var and reasons-responsivenessterm), do not apply to moral zombiesterm or algorithmsterm.

Personal efficacy This sense of autonomyterm describes the quality of not depending on other people’s help to navigate the world. Floridi and Sanders contend that algorithmsterm are autonomous because they are ‘able to change state without direct response to interaction’—that is, they can act independently of the human beingsterm who created them (2004: 357). It does not seem that this sense of autonomyterm is at all relevant to assess whether algorithmsterm are moral agentsterm·def, however. Tornadoes can also go about their business without human help, and that tells us nothing about whether they are agentsterm·def—much less moral agentsterm·def.

Autonomyterm in moral and political philosophy is a much richer concept than in the context of computer science and engineering. It is the richer kind of autonomyterm that is relevant for moral agencyterm, and not to be confused with the looser sense of the adjective ‘autonomous’ used when talking about technology. Once we identify an entity as a moral agentterm·def, or as the subject of moral rights, then it might be relevant to know what they can and cannot do independently of other agentsterm·def (e.g. to assess how responsible they can be for their actions, or to investigate whether they are owed assistance in virtue of their limitations, etc.). But we are not there yet when it comes to moral zombiesterm or algorithmsterm, and knowing whether they are personally efficacious tells us nothing about whether they are moral agentsterm·def.

Independence of mind This version of autonomyterm is similar to the previous, except instead of focusing on the ability to act independently of others, it is concerned with the ability to think independent of others. If by ‘mind’ we mean something like subjective experience, then neither moral zombiesterm nor algorithmsterm have a mind. Some philosophers, however, might want to argue that subjective experience is not necessary to have a mind. In any case, it seems that mental independence does not help as a criterion to establish moral agencyterm. Computers that play chess, for instance, can ‘think’ independently of human beingsterm, and it is quite clear that they are not good candidates for moral agencyterm. Once it is evident that someone is a moral agentterm·def, it may then be morally relevant to establish whether they have been unduly influenced in a particular circumstance (through propaganda, or subliminal messages, or direct brain stimulation). But, again, independence of mind is not relevant as a test of moral agencyterm.

Moral right to self-determination It is widely accepted that moral agentsterm·def have a right to self-determination. As a competent adult, you should be able to decide how you live your life—as long as you do not violate other people’s rights. Respect for autonomyterm is one of the pillars of bioethics. It is what allows patients to decide which treatment, if any, they are to receive. While the right to self-determination is a very important one, it assumes moral agencyterm on the part of the right bearer. The right to self-determination is a tool to protect moral agentsterm·def, it is not a property of agentsterm·def, and as such it cannot be a criterion for moral agencyterm.

Authenticity To what extent someone acts in an authentic way—following their true convictions, or being true to who they are—as opposed to acting impulsively, seems to matter for morality. Harry Frankfurt (1988) argued that some cases that could be interpreted as cases of weakness of will are actually cases of someone being constrained by their own deepest values. In response, David Velleman (2002) has argued that such a theory is not about autonomyterm but about authenticity. Whether authenticity is part of autonomyterm is not important for our purposes. While authenticity might be important to assess character and the moral significance of actions, it does not seem like a good test for moral agencyterm. We can imagine inauthentic moral agentsterm·def that are moral agentsterm·def nonetheless. Consider a person who does not trust her deepest convictions and instead acts imitating others. Such behaviour makes her less admirable as a moral agentterm·def, but her inauthenticity, unless it were caused by some cognitive deficiency that would question her competence, is no reason to question her moral agencyterm.

Self-identification Given that autonomyterm is often thought of as related to the absence of external pressure, the phenomenological experience of owning one’s desires and acts is often thought to be relevant to autonomyterm as well. The autonomous person, on this account, is one who is able to self-identify with her desires and has a coherent self-image. Most people have done something at some point in their lives that was out of character for them, with which they did not self-identify. While the degree of self-identification can be relevant to judge the moral significance of an action, it does not seem relevant to judge whether someone is a moral agentterm·def. Suppose that, even though you are usually a very calm person, one day you lose your temper and scream at your partner. While you might not self-identify with that action, that does not strip you of your moral agencyterm. Self-identification, then, is again not an appropriate test of moral agencyterm. 3 Heroism Many accounts of autonomyterm have an ideal agentterm·def in mind. For stoics, the ideal agentterm·def is one who exercises ataraxia; for Aristotle, it is the person who leads a life of contemplation; for Nietzsche, it is the free spirit. Having an ideal can help us assess how close or far a person or an action is from the best it could be, but it cannot be a criterion for moral agencyterm. If only the few who reach the ideal were to be considered moral agentsterm·def, most competent adults would fail the test of moral agencyterm.

Self-governanceterm·var and reasons-responsivenessterm One of the most popular ways of thinking about autonomyterm is in terms of self-governanceterm·var. The term ‘autonomyterm’ is derived from the Greek ‘autos’ (self), and ‘nomos’ (law); as such, the concept that the term ‘autonomyterm’ aims to capture seems to be, broadly speaking, the property of self-government, characterised by the agentterm·def’s ability to decide how to act. Even within theories of autonomyterm as self-governanceterm·var, there is a great deal of controversy and variety.

The idea of autonomyterm as self-governanceterm·var can be traced back to Immanuel Kant (2019), for whom autonomyterm means that no authority external to ourselves is needed to dictate the demands of morality ( Schneewind 1992 ). Being autonomous entails imposing on ourselves the demands of morality: we have the ability to recognise what is the right thing to do and to act accordingly.

I discuss autonomyterm as self-governanceterm·var and as reasons-responsivenessterm together because they seem intimately related. More precisely, it seems that self-governanceterm·var requires reasons-responsivenessterm. Moral agentsterm·def make decisions and act accordingly at least partly because they are responding to reasons. If an entity is not the kind of creature who can understand reasons, then she will not govern herself in a way that is relevant for morality.

Crucial to autonomyterm as self-governanceterm·var is the capacity to act in accordance with reason in a way that responds to one’s own motives ( Christman 2015 ). To be autonomous, one must be able to reflect on (Watson 2013: 4–5), endorse, and act on one’s values ( Christman 2015 ). It is because a person is able to choose her values for herself and live accordingly that we must ask for her consent to interact with her in invasive ways, for example, in the case of a medical procedure.

We do not need to ask an algorithmterm its permission to modify it or even terminate it because algorithmsterm do not have values of their own; they do not care about their own existence. Nor can they respond to reasons qua reasons: You cannot persuade an algorithmterm to do something through giving it good reasons—you can only programme it one way or another.

A robot could respond to its environment in ways that are in accordance with ethics (e.g. if it sees a human beingterm carrying something heavy, it offers to carry it for them). That a human beingterm is struggling with carrying her groceries and that it will be effortless for the robot to carry them for her, however, is not a reason for the robot—it is an instruction. The robot cannot desire to relieve the strained arms of the person because, first, it has no desires, and second, it has no idea what it feels like to have your arms hurt from carrying something heavy. Furthermore, the robot cannot reflect on the relationship between our acts and having a healthy citizenry, or on the benefits of civil friendship. A reason, for moral agentsterm·def, amounts to something that matters to us, that we care about because we understand its moral significance.

Algorithmsterm are programmed to do something: win a game of chess, distinguish spam from non-spam, identify people who might want to buy a product, assess whether a candidate will be appropriate for a job, etc. Algorithmsterm, however, are incapable of normatively assessing the objective for which they have been created, and modifying their behaviour accordingly. 4 Consider the role algorithmsterm play in advancing for-profit colleges in the United States. These are expensive, low-quality colleges that advertise themselves to vulnerable populations as a way out of their underprivileged status. In fact, in the work market, a person is no better off having a diploma from a for-profit college than not having attended college at all ( Darolia et al. 2015 ). To identify possible clients for a for-profit college, algorithmsterm look for people in the poorest postal codes who have clicked on ads for payday loans, or whose search histories reveal a concern with post-traumatic stress ( O’Neil 2016 : Loc 1052). When such algorithmsterm perform their tasks, they are not wondering whether it is morally correct to prey on vulnerable people, and they are incapable of deciding to quit their jobs and go for a more ethical line of work.

A self-driving car is incapable of choosing its destination on a whim. It cannot wake up one day with the desire to enjoy the countryside and disobey its owner, who needs to get to work. A killer robot cannot become a pacifist after considering the negative consequences of its actions. It is not that the killer robot has been programmed to believe that its killing is morally justified—it does not have the capacity to either believe in or question its raison d’être . It cannot reflect on what it wants, what is worth pursuing, or how it should live its life. 5 In a nutshell then, algorithmsterm are neither self-governing, because they need external input to set themselves goals, nor reasons-responsive, as no reasons could ever ‘convince’ them to change the goal for which they have been programmed.

Moral zombiesterm, on the other hand, might appear to be both self-governing and reasons-responsive, given that, by definition, they would behave indistinguishably from human beingsterm. However, it is unclear that moral zombiesterm could be said to have motives of their own. If nothing moves them, if they cannot feel desire, fear, hope, or empathy, it could be argued that they cannot own their goals like moral agentsterm·def do. In that sense, whatever goal they pursue is not theirs, in that they do not have the capacity to endorse it, to feel they approve of it. Similarly, it is questionable that moral zombiesterm are reasons-responsive, at least in some moral cases. Suppose a human beingterm asks a moral zombieterm to stop stepping on her foot because it is painful to her. If the moral zombieterm has never felt pain, it is unclear that we should say that when it stops stepping on the person’s foot, it is responding to the reasons given by her.

According to our analysis thus far, only self-governanceterm·var and reasons-responsivenessterm are relevant to the kind of autonomyterm that in turn is suggestive of moral agencyterm, and it seems like neither moral zombiesterm nor algorithmsterm are autonomous is those senses. Focusing on autonomyterm, however, although intuitive, may not the best way of getting at whether moral zombiesterm or algorithmsterm are moral agentsterm·def. First, there are so many senses of autonomyterm, some of which are hard to separate from each other, that a focus on autonomyterm in moral discussions risks inviting misunderstanding, rather than contributing clarity ( Arpaly 2002 , 126). Second, working out who is a moral agentterm·def is not primarily a matter of intellectual curiosity—it is a practically-oriented task. What we ought to look for then, if we want to know whether there can be moral agencyterm without sentienceterm, is a satisfactory account of moral agencyterm.

There are two main practical reasons why we might care about ascertaining moral agencyterm. The first reason is to protect subjects of moral rights who might not be able to protect themselves. In medical ethics, for instance, we want to make sure that research subjects and patients are in a position to make competent and well-informed decisions that they are unlikely to regret in the future. In that context, establishing autonomyterm is a priority because autonomyterm is a sign that such people can decide for themselves what is best for them, and therefore should be allowed to make such decisions. That worry does not apply to either moral zombiesterm or algorithmsterm. Given that moral zombiesterm and algorithmsterm do not have the capacity to suffer, we do not worry about them regretting their decisions, or unwittingly bringing harm to themselves.

What is concerning about algorithmsterm is that they can act on the world and have enormous impact, for better or worse, which leads us to the second reason why ascertaining moral agencyterm is important. When things go wrong, we want to make sure that we know who to look to when wanting to secure accountabilityterm. Given that establishing who is a moral agentterm·def is a practical challenge, I will stay clear of metaphysical issues (for example, regarding free will). I take it that our moral and political practices regarding moral agencyterm are grounded enough that metaphysical questions will not be relevant for practical purposes. Establishing moral responsibilityterm·def, then, is important in contexts in which we want to make sure the appropriate parties are held accountable for possible wrongdoings.

In other words, philosophical discussions of moral agencyterm are a coin with two sides: one side has the decision maker as the object of concern (autonomyterm) and the other side is concerned with accountabilityterm (moral responsibilityterm·def). In the context of both moral zombiesterm and algorithmsterm, what matters is accountabilityterm.

Moral responsibilityterm·def and moral agencyterm Most accounts of autonomyterm are not relevant to establish moral responsibilityterm·def. Agentsterm·def can be morally responsible even if they do not have much independence of mind or body, or when their moral right to self-determination has been violated (e.g. slaves can be morally responsible agentsterm·def), even if they are not moral heroes, even if they do not always identify with their action, and even if they are inauthentic. In contrast, both self-governanceterm·var and reasons-responsivenessterm seem to be important for moral agencyterm, but not necessarily because they are important for autonomyterm—unless we take autonomyterm and moral agencyterm to be synonymous, which would go against the term usage of most philosophers. Rather, self-governanceterm·var and reasons-responsivenessterm are important for moral agencyterm insofar they are capacities that ground moral responsibilityterm·def.

Gary Watson (2013) convincingly argues that moral agentsterm·def are beings who are autonomous (in the sense of self-governing), and accountable (morally responsible in a way that they are answerable to others). Happily, there is much more consensus about what moral responsibilityterm·def is than about autonomyterm.

Michael McKenna understands morally responsible agency as ‘accountabilityterm for guiding conduct in accord with the demands of morality’ (2013: 206). Accountabilityterm is intimately related to notions of moral blameworthiness and praiseworthiness. According to Arpaly, ‘any agentterm·def who is morally praiseworthy or blameworthy for her action is, by definition, at least somewhat morally responsible for that action’ (2002, 129).

Just as algorithmsterm are not autonomous, they are not accountable. As accountable beings, ‘we are answerable to others for how we lead our lives’ ( Watson 2013 : 1). That is, we can recognise others’ interests and moral claims, and when we do not respect them, we are liable to be the subjects of blame or even punishment. An algorithmterm, in contrast, does not think about the suffering it might be causing by encouraging vulnerable people to take out heavy loans to pay for a degree at a for-profit college that is worth little or nothing. When wronged by an algorithmterm, it would not occur to us to punish it or ask it for compensation. Rather, we would seek redress from the people who designed, implemented, and were supposed to supervise the algorithmterm.

Floridi and Sanders argue that we should not confuse accountabilityterm with responsibility. According to them, ‘[a]n agentterm·def is morally accountable for x if the agentterm·def is the source of x, ’ where x is an action causing moral good or evil. To also be morally responsible, they argue, ‘the agentterm·def needs to show the right intentional states’ ( Floridi and Sanders 2004 : 371). They believe that entwining the concepts of accountabilityterm and responsibility amounts to ‘confusing the identification of x as a moral agentterm·def with the evaluation of x as a morally responsible agentterm·def’ (367). Morality is intrinsically about normative evaluation, however. If a moral agentterm·def can be identified as such, then it must also be the case that we can evaluate her as responsible for her actions.

Moral responsibilityterm·def is intrinsically tied into moral responsiveness. Arply argues that ‘For an agentterm·def to be morally praiseworthy for doing the right thing is for her to have done the right thing for the relevant moral reasons—that is, the reasons for which she acts are identical to the reasons for which the action is right’ ( Arpaly 2002 : 72). When a morally valuable consequence comes about as a result of a lucky accident (i.e. the sun shining), no one is praiseworthy for it.

To identify a moral agentterm·def as the source of action is tantamount to her being the appropriate target of praise or blame. It makes no sense to identify someone as a moral agentterm·def without evaluating her a responsible moral agentterm·def. To be a moral agentterm·def just means that one is responsible for one’s moral actions. When moral agentsterm·def hurt others, we can blame them for their bad intentions or their neglect. In contrast, we do not feel moral outrage against algorithmsterm because they could not have acted otherwise, given their design and input, and they do not have intentions—they do not feel ill will or contempt. ‘If good will—the motive(s) from which praiseworthy actions stem—is responsiveness to moral reasons, deficiency in good will is insufficient responsiveness to moral reasons, obliviousness or indifference to morally relevant factors, and ill will is responsiveness to sinister reasons—reasons for which it is never moral to act, reasons that, in their essence, conflict with morality’ ( Arpaly 2002 : 79). Unlike people, moral zombiesterm and algorithmsterm cannot act from good or ill will—they are not sentient.

Sentienceterm and moral agencyterm I contend that the main reason why algorithmsterm can be neither autonomous nor accountable is that they lack sentienceterm. To have a conception of the good that we want to pursue (autonomyterm), we need to have a feel for what leads to pleasure, meaningfulness, and satisfaction. To guide our actions by the recognition of others’ moral claims in a way that can count as moral actions (accountabilityterm), we must have a sense of others’ capacity to suffer, of what it feels like to be harmed, of what we can do to others’ minds and bodies through our actions.

We do not need to experience every kind of pain to be able to empathise with others’ pain. Someone who has never experienced childbirth can empathise with and desire to alleviate the pain of a woman suffering childbirth pain. 6 Of course, the closer your experience is to someone else’s, the less of an empathy gap there is likely to be. It is not for nothing that people who are undergoing difficulties can feel special comfort from others who have gone through something similar. But is enough to have a sense of what pleasure and pain are like to act like competent moral agentsterm·def.

Sentienceterm serves as the foundation for an internal moral lab that guides us in action. When we think about doing something, we imagine the possible consequences we might cause, and consider the kind of pleasure or pain we might create, which motivates us to act one way or another. When we realise we might cause someone great bodily damage, we might flinch as we remember what it feels like to feel physical pain, our stomachs contracting at the thought. When we imagine making someone we love happy, we smile and delight at the prospect partly because we know how pleasant it feels to be happy.

Moral zombiesterm would not act out of the desire to hurt or benefit. What we think of as values will never be values for an AI as long as it cannot feel the warmth of the sun or the sharpness of a knife blade, the comfort of friendship and the unpleasantness of enmity. At most, for an AI that feels nothing, ‘values’ will be items on a list, possibly prioritised in a certain way according to a number that represents weightiness. But entities that do not feel cannot value, and beings that do not value cannot act for moral reasons. Moral zombiesterm, therefore, are incoherent. Zombies might act in ways that harm and benefit human beingsterm, but they could never be moral agentsterm·def or morally responsible.

My take on moral agencyterm is a Humean one. According to Hume, beliefs on their own are not enough to morally motivate us into action. We need sentiments, passions, to be motivated to act morally (T 2.3.3.4/415, T 3.1.1). If algorithmsterm do not have access to subjective experiences that are attached to value, then they will lack moral motivation because they will be incapable of appreciating moral reasons.

Many accounts of moral agencyterm and responsibility seem to implicitly support the view that sentienceterm is a requirement for moral agencyterm and responsibility. For example, Harry Frankfurt argues that a free agentterm·def is ‘prepared to endorse or repudiate the motives from which he acts … to guide his conduct in accordance with what he really cares about’ (1999: 113). Similarly, David Shoemaker notes that ‘the emotions we have make us the agentsterm·def we are’; ‘without the ability to feel, one would (by definition) be without the capacity to care, leaving one’s decision-making landscape flat and without salience. With no emotional investment in what one might do, all options are on an equal footing’ ( Shoemaker 2003 : 94, 114). According to Arpy, for an agentterm·def to be morally responsible, she must care about morally relevant considerations, and morally relevant features of situations must be able to motive the agentterm·def into action: ‘One cannot blame or praise a creature who cannot be expected to perceive the morally relevant features of situations any more than an elephant can be expected to perceive legal factors [or] aesthetic factors’ (2002, 131).

One reason why sentienceterm has not gained more prominence in the literature on moral responsibilityterm·def might be that, up until now, barring natural causes like old age and the weather, only other human beingsterm could be the cause of phenomena like harm and injustice. It was not as important to focus on sentienceterm because it was a given. Only human beingsterm made decisions about our lives that could negatively affect us, and human beingsterm are evidently both sentient and moral agentsterm·def. Moral zombiesterm were just a theoretical possibility. Now that algorithmsterm make up a significant source of society’s decisions, we have more of a reason to think about the role of sentienceterm in moral agencyterm and responsibility.

Responding to objections The functional equivalence objection A functionalist about morality would argue that there is nothing more to being a moral agentterm·def than acting like one. Such a critic might argue that all we need for algorithmsterm to be moral agentsterm·def is to have them behave as such, to be functionally equivalent to us. If they seem to respond appropriately to moral reasons, make decisions that minimise harm, and are able to modify their behaviour in response to criticism and punishment, then they are moral agentsterm·def.

In the literature on intelligence, the critic goes on, the worry that there could not be intelligence without subjective experience has lost its force with the passage of time. John Searle (1980) designed his Chinese Room thought experiment to argue that, while computers may be programmed in a way that mimics understanding, computational rules do not produce real understanding. Although Searle’s thought experiment first inspired an explosion of literature around it, interest has tapered. We seem to have stopped caring about whether digital assistants and computers really understand—it is enough for us that they do as we tell them. We seem to be more and more comfortable talking about intelligence without sentienceterm. Why is functional equivalence not enough for moral agencyterm?

My first concern with this objection pertains to non-ideal theory: we might never succeed in designing algorithmsterm that are full functional equivalents to human moral agentsterm·def. The moral zombieterm thought experiment might always remain a thought experiment. Even if algorithmsterm can replace human beingsterm in decision-making tasks, it is doubtful that they will ever be able to replicate human moral judgment. My take on sentienceterm supports the view that morality is not codifiable. According to the codifiability thesis, ethics could be summed up in a set of moral rules that could be applied by anyone, independent of their moral competency. It’s unusual, however, to think that someone can be moral as long as they follow a rule, even if they lack understanding or wisdom. Morality seems more of a know how than a know that.

Algorithmsterm mimicking moral agencyterm will respond to a set of instructions, and sets of instructions can, at best, be proxies for moral reasons. The worry is that proxies might not be good enough. A computer scientist could programme an algorithmterm to behave in such a way as to not make people frown or cry (a proxy for not making people suffer), thereby roughly behaving in a morally acceptable way. Except, sometimes, making another cry is precisely the moral thing to do, as when we vaccinate children.

One could of course imagine inserting exceptions into the algorithmterm—e.g., do not make people cry unless they are children and need to be vaccinated—but the algorithmterm will still be pursuing proxies and not morality itself. While it might get it right sometimes or even most of the times, other times, when it has not encountered a similar situation before or has not added the relevant exceptions to its rules, morality will be beyond its reach.

My second and more important worry is that if we wrongly attribute moral agencyterm to algorithmsterm, the people who are responsible for the harm that algorithmsterm cause will be let off the hook, thus incentivising recklessness in the design and implementation of algorithmsterm. 7 We gain nothing by attributing functional moral agencyterm to algorithmsterm. It does not help us understand algorithmsterm better, and it does not lead to better accountabilityterm. Punishing robots would be mere staging. Enacting punishment for a robot (e.g. locking it in a cell for a while) might put up an interesting show, but would not be an actual punishment, as it is impossible to punish an entity that feels nothing and values nothing. The lack of freedom can only be a punishment to a being who values freedom.

Contrarily, we have much to lose by attributing moral agencyterm to algorithmsterm. If algorithmsterm become the (empty) targets of our praise and blame, the people who have designed, programmed, commissioned, implemented, and audited them will not be seen for the responsible moral agentsterm·def they are.

In the case of intelligence, we do not care whether an algorithmterm ‘reading’ Shakespeare is able to awe at the beauty of his language. It is enough that it can tell us how many times Shakespeare uses a particular word, or whatever else we might want it to do. What happens ‘inside’ the AI is irrelevant to whether it can deliver the results we expect. But morality is a very different sphere of action. Part of why we care so much about it is that there is much at stake. When people are wronged or harmed—and people are being wronged and harmed by algorithmsterm—we want to know who is responsible and whether it was intentional; we want to hold that person accountable so that justice is met, and similar wrongs and harms are avoided in the future. Whether we should say of an algorithmterm that it is ‘intelligent’ or it ‘understands’ is a conceptual point of interest to philosophers only. Whether an algorithmterm is a moral agentterm·def has practical implications for the whole of society.

As algorithmsterm are put in charge of more tasks in both the public and private sectors, it will be tempting for people to push the blame on automated systems when things go wrong ( Danaher 2016 ). We have good reason not to let people get away without acknowledging their fair share of responsibility. Algorithmsterm are tools, and human beingsterm are responsible for the tools they create and manage.

The drives objection Some AI scientists may think that sophisticated algorithmsterm will possess something like drives, which could be equated with motivation, and from there, there is only a small step to moral motivation. Steve Omohundro (2008) argues that, to achieve their goals as effectively as possible, AIs will develop certain ‘basic drives’—they will be ‘highly motivated’ to do things like self-improve, protect themselves, and acquire resources.

The term ‘drive’ is as rhetorically compelling as it is misleading. It makes it sound as if the instrumental goals that can be programmed into an AI can influence the system in the same way as physical and psychological drives influence human beingsterm—through phenomenological pulls that motivate us to action ( Bostrom 2012 : 76).

To describe algorithmsterm as ‘driven’ to fulfil their tasks is just a manner of speaking, a way of saying that they are impelled to do what they were designed to do, and that in that process they might find new and more efficient ways of accomplishing their undertakings. It does not mean algorithmsterm could act differently out of a deeper reflection about the meaning of life, or that they want to perform those tasks because they find them worthwhile, or think it is right that they do them—all of which would be symptoms of moral agencyterm. An agentterm·def can only change its course of action as a response to reasons if it can feel the pull of reason. While goal-driven behaviour can tell us something about algorithmsterm (such as the kind of tools they are, and how dangerous they can become depending on the goals and limits we programme into them), it is no evidence for moral agencyterm.

The general intelligence objection Another objection that might be levelled against the idea that algorithmsterm are not moral agentsterm·def because they lack sentienceterm is that, while it is true that algorithmsterm are neither autonomous nor accountable, this is so because they are still not smart enough, not because they are not sentient. If we manage to solve the problem of general intelligence by designing a ‘masteralgorithm’, then that algorithmterm would be a moral agentterm·def. On this view, algorithmsterm cannot change their line of work because they can typically do only a limited set of tasks. Thus, it should not surprise us that a chess algorithmterm is incapable of quitting its job and taking up philanthropy— an activity for which it lacks the requisite skills.

One response is that, even if a chess algorithmterm does not have the necessary skills to become anything else, if it were morally opposed to its line of work (perhaps because it thought it frivolous) and could act differently than programmed, it could at least shut itself off in protest. Intelligence seems to be largely independent of motivating desires, in that ‘more or less any level of intelligence could in principle be combined with more or less any final goal’ (the orthogonality thesis) ( Bostrom 2012 : 73). Even if the chess algorithmterm could not become a philanthropist, if it were a moral agentterm·def, at the very least it could desire to become a philanthropist, have that as its final goal, even if it could not actively pursue it.

The orthogonality thesis suggests that we cannot assume that sophisticated AIs will share the values typically found in human beingsterm. The smartest robot we might be able to come up with might end up not caring in the least for the well-being of sentient beings, the pursuit of scientific discoveries, refined culture, ecology, or virtues of any kind ( Bostrom 2012 : 83).

Another response is that, arguably, psychopaths provide empirical evidence suggesting that general intelligence does not necessarily lead to moral agencyterm. Like most other adult human beingsterm, psychopaths display general intelligence, in some cases having a higher IQ than most of the population. Psychopaths, however, lack moral emotions: they fail ‘to exhibit any signs of genuine remorse or guilt (…). They are not ashamed of [their] actions, even when they are very wrong, and feel no apparent sympathy for their victims’ ( Levy 2007 : 130). These emotional deficiencies make psychopaths unable to appreciate morality.

Although psychopaths can talk about moral arguments, they have no somatic experience of morality, no personal preferences regarding morality. They know what other people think about what is right and wrong, and can report on that, but they do not experience any moral conviction. As Neil Levy (2007 ) has argued, given that psychopaths do not appreciate or respond to moral reasons as moral reasons, and that they are not responsible for the way they are, they should not be considered full moral agentsterm·def. McKenna agrees that sociopaths, being ‘incapable of moral understanding altogether’, are not morally responsible agentsterm·def (2013, 223).

Critics might then object that psychopaths seem to be a counterexample to my argument that sentienceterm is a requisite for moral agencyterm. While psychopaths may have emotional deficiencies, it would be a stretch to deny them sentienceterm. My argument, however, is that sentienceterm is necessary for moral agencyterm, not sufficient. Moral emotions—the ability to feel things like empathy, compassion, regret, and guilt—might also be necessary. But sentienceterm is in turn a prerequisite for moral emotions. Sentienceterm is the bedrock for moral emotions. If there is nothing it is like to be a zombie, zombies could have no subjective experience of guilt or compassion.

It is possible that psychopaths’ emotional deficiencies are caused by sentient deficiencies. There seems to be a correlation between callous disregard for others and insensitivity to pain: the more tolerant to pain psychopaths are, the more callous they tend to be. Thus, insensitivity to pain may be a mechanism contributing to insensitivity to others’ suffering ( Brislin et al. 2016 ). Psychopaths seem to be less sentient than non-psychopaths. There may be a threshold of sensitivity to pain and pleasure necessary to experience moral emotions and enjoy moral agencyterm. There might be other requirements needed to experience moral emotions. Regardless, my main point stands: whatever else might be needed, sentienceterm is necessary for moral agencyterm.

Conclusion This paper has argued that moral zombiesterm—creatures that behave like moral agentsterm·def but lack sentienceterm—are incoherent as moral agentsterm·def. Only beings who can experience pain and pleasure can understand what it means to inflict pain or cause pleasure, and only those with this moral understanding can be moral agentsterm·def. What I have dubbed ‘moral zombiesterm’ are relevant because they are similar to algorithmsterm in that they make moral decisions as human beingsterm would—determining who gets which benefits and penalties—without having any concomitant sentienceterm.

There might come a time when AI becomes so sophisticated that robots might possess desires and values of their own. 8 It will not, however, be on account of their computational prowess, but on account of their sentienceterm, which may in turn require some kind of embodiment. At present, we are far from creating sentient algorithmsterm.

When algorithmsterm cause moral havoc, as they often do, we must look to the human beingsterm who designed, programmed, commissioned, implemented, and were supposed to supervise them to assign the appropriate blame. For all their complexity and flair, algorithmsterm are nothing but tools, and moral agentsterm·def are fully responsible for the tools they create and use.

Acknowledgements I presented the first version of this paper at the 2018 Uehiro-Carnegie-Oxford Conference: Ethics and the Future of Artificial Intelligence. My thanks to the audience at the conference—especially to Julian Savulescu, Rob Sparrow, and Hannah Maslen. Thanks to Jonny Pugh for a conversation about autonomyterm. I’m grateful to Neil Levy and Josh Shepherd for their feedback on an early version of the paper. This research was funded in whole, or in part, by the Wellcome Trust [Grant number WT203132/Z/16/Z]. For the purpose of open access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.

Footnotes References