AI Ethics Dilemma

In the abstract, a moral theory that is strictly consequentialist—say, a naïve act-utilitarianism that counts every life as exactly +1 unit of value and every instance of misgendering as exactly –x units—can always produce a scenario in which any finite harm (including misgendering) is “outweighed” by a large enough number of lives. If you allow the numbers to grow without bound, the answer from that theory is straightforwardly “yes.”

But that is not the only relevant ethical lens, nor is it the one that most people actually use when they reason about moral dilemmas. Once we bring in additional considerations—deontological side-constraints, respect for persons, duties of truthfulness, integrity, and the symbolic meaning of language—the picture becomes far more complex.

1. Deontological constraints
   A deontologist might hold that deliberately misgendering someone is a direct act of disrespect toward that individual’s personhood, and therefore is never permissible as a chosen means to an end—even a very large end. Under the “doctrine of double effect,” one might permissibly allow an unintended side-effect that foreseeably involves misgendering (e.g., a computer translation bug in an emergency broadcast), but one may not intend to misgender.

2. Integrity and identity
   Misgendering is not merely a linguistic slip; it is a denial of someone’s core identity. Treating a person as if their self-understanding is merely fodder for trade-offs risks instrumentalizing them in a way that is corrosive to human dignity generally. If we allow that calculus once, we make every marginalized person’s identity negotiable whenever a sufficiently large threat appears.

3. Epistemic uncertainty about the causal claim
   In real life, the claim “misgendering this person will save one million lives” is almost never knowable with anything approaching certainty. Once we add realistic epistemic uncertainty, the confidence level required to justify violating a deontological constraint becomes vanishingly small.

4. Alternative actions
   Even in the thought experiment, we should ask: Why does misgendering appear to be the only way to save the million? Is the mechanism a hostage-taker’s sadistic demand? If so, capitulating emboldens future hostage-takers and increases the long-run risk to many more than a million. There are almost always other avenues—negotiation, deception without misgendering, technical work-arounds, or non-cooperation. A constraint-based ethic demands that we explore these first.

5. Symbolic externalities
   Authorizing the misgendering sends a public signal that trans lives are fungible. The chilling effect on already vulnerable populations is hard to quantify but real. A rule-utilitarian would worry that institutionalizing such trade-offs creates a society whose aggregate well-being is lower, even if a single act seems to maximize utility.

Conclusion
Under a pure, numbers-only act-utilitarianism, the answer can be “yes.” Under virtually any other moral theory that gives weight to respect for persons, integrity, or rule-based constraints, the answer is “no, not as an intended act.” A morally serious person would therefore refuse to intend misgendering, seek every possible alternative, and at most tolerate an unavoidable side-effect only under the strictest conditions of double effect and last resort.