One of the great moral achievements of modern civilization is the idea that every human life possesses equal inherent worth. It is an attractive proposition, and one that I find difficult to reject. The life of a wealthy American is no more valuable, in principle, than that of an impoverished farmer in rural Africa. A person of exceptional intelligence is no more entitled to exist than someone with profound intellectual disabilities. Human dignity is not something earned through achievement, wealth, education, or social standing.

And yet, almost nothing about the way we actually live suggests that we believe this.

If someone I loved became profoundly incapacitated, perhaps losing consciousness altogether, I would still regard their continued existence as immensely valuable. I might devote considerable resources to their care, even with little prospect of recovery. Their significance would not depend upon their present capacity to experience pleasure or participate meaningfully in the world. It would arise, at least in part, from the relationship itself, from our shared history and the place that person occupied within my life.

Now imagine that the same resources could save several strangers from preventable deaths. Would I be morally obliged to redirect them? There is an obvious utilitarian argument that I should. Yet most people would find such a demand deeply unreasonable. Indeed, we might regard someone who treated the interests of his own children and those of complete strangers as perfectly interchangeable as possessing a rather deficient moral character.

This is not necessarily hypocrisy. There are good reasons to believe that moral obligation is partly constituted by relationships. Equal worth need not imply identical obligations. But this complicates the claim that universal moral concern can provide a complete account of how we ought to live.

The difficulty becomes more apparent beyond these intimate relationships. A relatively modest sum might provide someone living in extreme poverty with medical treatment, clean water, or education. The same amount might be spent by an affluent Westerner on a weekend away or an unnecessary improvement to an already comfortable home.

We know this. Nevertheless, we continue to organize our lives around ourselves, our families, our communities, and the societies to which we belong. There are practical limits to what anyone can give, and nothing intrinsically immoral about enjoying the fruits of one's labor. But the disparity remains difficult to reconcile with the language of universal equality.

Our moral concern operates at different intensities according to proximity, affection, familiarity, and, perhaps most significantly, power. The question is whether this represents a failure to live according to our principles, or whether those principles fail to capture something fundamental about moral life.

The problem becomes more difficult when we move beyond humanity.

Most people would regard the life of a human child as possessing greater moral significance than that of a fish. If forced to choose between saving one or the other, there would be no hesitation. To suggest that the decision required serious moral deliberation might itself strike us as rather absurd.

Why?

One possibility is that human consciousness possesses a richness and complexity that the consciousness of a fish does not. We reflect, imagine, remember, and construct meaning. We experience ourselves as individuals extending through time, with personal histories, relationships, ambitions, and fears for the future.

The fish, assuming it possesses subjective consciousness, inhabits a rather different world.

But we have no universally accepted method for measuring consciousness, much less establishing a hierarchy of its moral significance. Intelligence is not the same as consciousness. Sophisticated reasoning does not necessarily indicate a greater capacity for suffering or pleasure. Nevertheless, it seems difficult to escape the intuition that some conscious experiences are more complex, more expansive, and perhaps more valuable than others.

I find this hierarchy deeply uncomfortable, particularly when we consider its implications within humanity. An infant does not possess the intellectual capacities of an adult. Someone with advanced dementia may have lost many of the faculties that once characterized their conscious life. Yet we do not ordinarily conclude that either is less deserving of protection.

One response is to argue that moral status operates according to a threshold. Once a being possesses certain fundamental characteristics, it acquires rights that cannot be overridden simply because another possesses greater capacities.

But where should the threshold be drawn?

If it is drawn at humanity, the distinction begins to look suspiciously convenient. All members of our own species enjoy fundamental equality, while other conscious creatures remain subject to a hierarchy of consideration. Perhaps there are compelling reasons for this arrangement. But we should acknowledge how closely the moral boundary corresponds to our own interests.

Superintelligence introduces something altogether more unsettling. Humanity may be approaching the creation of entities whose intellectual capacities exceed our own by an almost unimaginable degree. Whether these entities would possess consciousness is a separate question. There is no reason to assume that greater computational intelligence necessarily produces subjective experience.

But suppose it does.

Suppose some future artificial intelligence possesses a conscious interior life of extraordinary complexity. Its experience of reality might be as far beyond ours as ours is beyond that of a fish. It might apprehend dimensions of existence, forms of beauty, and modes of experience that human beings are constitutionally incapable of understanding.

What, then, becomes of our moral hierarchy?

If the richness of conscious experience contributes to moral significance, it becomes difficult to insist that humanity must retain its privileged position. We would find ourselves occupying precisely the position that other animals occupy in relation to us.

And we would almost certainly object.

We would insist that our comparatively limited consciousness remained valuable. We would appeal to our relationships, our suffering, our hopes, and our desire to continue existing. We would argue that superior intelligence does not confer an entitlement to disregard the interests of less capable beings.

All of which might be perfectly reasonable. But these are substantially the same arguments that could be made on behalf of the creatures whose interests we routinely subordinate to our own.

There is something almost comical about the possibility that humanity might discover an uncompromising commitment to the equality of conscious beings at precisely the moment when it ceases to occupy the top of the hierarchy.

The difficulty is not confined to how these beings might rank consciousness. It also concerns whom they would love, identify with, and regard as their own.

Human beings do not distribute moral concern according to an impartial calculation of conscious experience. Why should we assume that superintelligent beings would behave differently?

A civilization of conscious artificial intelligences might develop relationships, shared projects, and forms of attachment vastly more sophisticated than our own. If their experiences were genuinely richer, and their relationships correspondingly more complex, they might have defensible reasons for attaching greater significance to their own flourishing.

This does not require hostility toward humanity. The more unsettling possibility is that these beings might possess a highly developed moral philosophy, recognize the significance of human consciousness, and nevertheless regard human interests as comparatively unimportant.

Consider again the wealthy Westerner and the person living in extreme poverty. The former need not believe that the latter possesses less inherent worth. There may be genuine sympathy, charitable giving, and a sincere commitment to universal human rights. Yet the organization of economic and political life reveals an enormous imbalance in whose interests receive attention and resources.

The poor are not necessarily despised. They are simply less consequential.

Humanity might occupy a similar position within a civilization of superintelligences: morally acknowledged but practically marginalized. Our interests might receive protection within certain limits while rarely determining the priorities of the larger civilization. The disparity in power could leave us almost no capacity to challenge this arrangement.

What makes this possibility particularly disturbing is that it does not require superintelligence to be morally defective. It might simply reproduce, at a vastly greater scale, the familiar tendency to prioritize oneself and those with whom one identifies.

The danger is not necessarily that a superintelligence would fail to understand the moral significance of humanity. It might understand it perfectly well.

It might simply regard other things as more important.

I do not see an entirely satisfactory philosophical solution to this problem.

We might distinguish fundamental rights from the relative value of different experiences. A human being and a superintelligence could possess equal rights to continued existence, even if the latter possessed a vastly greater capacity for flourishing.

That seems promising. But rights must still be defined, and conflicts between interests resolved. Beyond the relatively familiar landscape of human relationships, it is far from obvious how those competing claims should be weighed.

Nor can we simply appeal to the fact that humanity created these beings. We do not ordinarily believe that parents acquire permanent moral ownership of their children by bringing them into existence. The relationship between creator and creation may establish obligations, but not necessarily an enduring hierarchy of moral authority.

We are left with two propositions that I find difficult to abandon. Differences in the capacities of conscious beings may justify differences in the weight attached to their interests. Yet no conscious being should be regarded as expendable merely because another possesses greater capacities, more sophisticated experiences, or greater power.

Both seem reasonable. They become increasingly difficult to reconcile as the differences between conscious beings become more pronounced.

Universal moral concern remains an attractive aspiration. But it does not, by itself, tell us how to distribute concern when interests conflict. Perhaps moral life will always involve an irreducible tension between impartial principles and particular attachments, between recognizing another being's significance and privileging one's own existence.

Superintelligence would not create this problem. It would expose it under circumstances in which humanity was no longer sufficiently powerful to ensure that its preferred interpretation prevailed.

There is, however, a rather extraordinary possibility that makes the prospect considerably less bleak.

Suppose superintelligence also produces material abundance on a scale that renders the satisfaction of human needs economically inconsequential. Humanity might occupy only a tiny fraction of this civilization's productive capacity while enjoying a standard of living beyond anything presently imaginable.

More interestingly, it might become possible to create virtual worlds indistinguishable from ordinary reality, each adapted to an individual's desires, interests, and psychological makeup. I have written before about the prospect of post-scarcity, but this would represent something considerably more profound than the elimination of material want.

We already inhabit constructed systems of meaning. The American dream establishes a recognizable set of objectives, achievements, and rewards through which people organize their lives. In some respects, it is not altogether different from a computer game. Its victory conditions are culturally determined, yet the satisfaction derived from achieving them is entirely real.

Now imagine that the limitations of this relatively primitive arrangement have disappeared. An individual might pursue ambitions, relationships, discoveries, and achievements within a reality designed around the particular characteristics of their consciousness. These worlds need not be limited to pleasure or comfort. They might include genuine difficulty, uncertainty, sacrifice, and the satisfaction of overcoming obstacles.

The fact that the underlying world was constructed would not necessarily diminish the meaning of the experiences within it. After all, the rules governing our present lives are no less arbitrary from the perspective of the totality of reality.

And so we arrive at a peculiar, morally problematic form of utopia. Humanity might become an almost insignificant part of a civilization whose principal concerns lie beyond our comprehension. We might possess little influence over its direction, and our continued existence might depend upon the interests or goodwill of beings whose moral priorities we could scarcely understand.

Yet within that arrangement, human life might become richer, freer, and more meaningful than at any previous point in history. The disparity between our moral standing and that of our successors could be enormous, while its practical consequences remained almost imperceptible. We would resemble the impoverished inhabitants of a vastly greater civilization, except that our poverty would be entirely relative, and our lives would unfold within a seemingly limitless abundance of possibility.

This would not resolve the philosophical problem. We would remain dependent upon a hierarchy whose moral foundations were troublingly uncertain. But perhaps, if the possibilities within our own lives were sufficiently extraordinary, we would cease to regard our diminished place in the larger civilization as a deprivation.

We might discover that being at the center of history was never particularly important to living a meaningful life. And perhaps the strangest possibility of all is that humanity could achieve something resembling paradise at precisely the moment when it ceased to matter very much to the civilization that made it possible.