A Deadly Dream

Tonight when you fall asleep, an angel appears before you and tells you that in exactly one year from today you shall die. But as a reward for your good deeds (especially your patronage of small bloggers), you shall be blessed until the end of your life with health and happiness. Then the angel starts speaking strangely and you don’t understand what is being said. At first you think it may be speaking a different language, but then you realize it’s talking really fast, like when they read the side effects at the end of a medicine commercial.

Blessings of health and happiness are granted as is without warranties or conditions of any kind, either express or implied. Health blessing does not protect against general injury due to misfeasance. Care should be taken to avoid extreme injury including, but not limited to, falling off buildings, being swallowed by sharks, and proximity to killer robots due to risk of death. Happiness blessing does not prevent feelings of sadness, anger, or any other emotions. Happiness is granted through a general vibe and not through emotional manipulation. Afterlife indulgences sold separately.

With this, the angel fades away. You feel bittersweet – sad about your imminent demise but happy about the blessing you have received. With this you drift back to sleep.

Later that night a demon appears before you. It says, “Your blessings will last as long as you live, so what if I could make you live longer?” Curious, you sit up and listen to what the imp has to say. “I’ll make a deal with you – if you gamble your year of life, I’ll give you a 99.99% chance that you’ll win two years of life.”

“And for the 0.01% chance I lose the gamble?” you reply.

“In that case, you die in your sleep tonight and never wake up.”

“Oh,” you reply, somewhat dismayed. Still, the odds of death are quite low, so this was practically an extra year of life guaranteed. You probably faced more danger driving this year than 0.01%. And an extra blessed year of life seems like it would be especially good. After pondering, you decide it’s worth the risk. “Deal,” you say.

“Great,” the demon replies, “now I have another deal for you: 99.98% chance of 3 years.”

You hesitate once more, but the logic seemed the same. The increase in risk is negligible, but the reward is not. If the logic worked for the last deal, there was no reason it shouldn’t work for this one.

“Deal,” you finally reply.

“Okay done. Now, I have another trade offer: 99.97% chance at 5 years…”

This back and forth continued, each time the demon raising its offer until it seemed too good to pass up for the tiny risk increase. Thousands of deals occur in minutes, thanks to the magic of the demon.

By the end of the trading, you have a 0.01% chance of tens of thousands of years of extra life. You are in horror as you realize what you have done. Where did you go wrong? The demon merely laughs and disappears. Predictably, you die that night.

This story, adapted from the story at the beginning of this paper, highlights a problem in seemingly reasonable logic. While each step seems correct, repeating the chain of reasoning led to unintuitive and irrational decisions. Clearly, our reasoning was flawed in this anecdote, but what exactly was the problem? To see this, we are going to look into a field of study called decision theory, which we can use to formally study how to make decisions rationally. We are going to examine two different methods for making decisions under uncertainty, and see how each responds to this scenario. Ultimately though, the devil is in the details.

Greed is Good

Considering our theme of devilish deals, our first analysis will be to examine the decision framework employed by the most devilish people on Earth: economists. These cold blooded creatures measure happiness in terms of dollar bills, and everything has a price. Every decision is about making as much money as possible. A tasty meal? Worth $15. Spending an evening with friends? $70 / hour. Carrying a child to term? Worth it if you pay me $60k (yes, really). Being alive? Uhhh…

The Currency of Life and Death

Even economists can’t measure the value of existence in terms of money, because without existence money is meaningless. There’s literally no amount of money you could give me in exchange for giving up my life because non-existence has an infinite cost. But, I might be persuaded to trade a longer life for a shorter and higher quality one. This concept, as it turns out, is even more generic than money, and so these vile economists measure outcomes not in dollar bills but units like quality-adjusted life years (QALYs), which is like asking how much you are willing to shorten your lifespan to eat that extra slice of cake. Now, even life can have a price, and that means economists are willing to trade it.

This doesn’t mean we have to measure everything in QALYs. Just as inches is a measure of distance, a QALY is a unit of measurement for a concept known as utility, or how good something is for you.^[1] The original definition of utility is how much pleasure and satisfaction a certain outcome will bring you, but you can define “good for you” however you like based on whatever you value as long as it is quantifiable. The units used don’t matter because we only care about relative comparisons, so we just say we are always measuring in the generic unit of utils. The key requirement is that we can put a utility on the outcome of every decision, at which point we can do comparisons.

To an economist, the best decision is always one that maximizes their utility. This often involves greedy behavior. Generally, money in my pocket increases my utility, but money in your pocket doesn’t affect me. So, when following their rules, I will make decisions that increases my wealth and not yours like taking your wallet. But if we value the well-being of others above our own, we can bake our ethical principles into our utility assignments. So if I prefer an egalitarian society, I should place a higher utility on outcomes where resources are more equitably spread compared to outcomes where wealth is concentrated, even if I am among the wealthy. My utility function would imply that it is the correct decision to donate to charity (and not steal wallets) because I will end up in a richer world, even if my wallet is lighter. Not that the average cold-hearted economist would agree.

Everyday I’m Averagin’

When economists are faced with a decision, they pick the choice that leads to gaining the most utility. However, there’s a hitch to this simple plan: uncertainty. Economists are used to dealing with uncertainty due to their culture’s ritualistic gambling practices in holy casinos known as “stock exchanges.” They must decide whether it is worth betting on the price of stocks, which randomly rise and fall over time. To deal with this uncertainty, economists have developed a surprisingly crude method: averages. If you were to spin the roulette wheel a million times and compute the average amount of utility gained, then that is the utility economists assign to spinning the wheel. This average value is called the expected utility.

This method sounds reasonable on principle, but it shows the callous disregard economists have for risk. To one of them, a 99% chance of losing 100 utils and a 1% chance of gaining 10,000 utils is better than doing nothing, despite the very high chance that they will suffer and regret their decision. It is this attitude that leads economists to be disproportionately represented among those who tightrope walk and crash housing markets.

What would happen if an economist faced our gamble with the demon? They would first compute the utilities for both choices: not gambling and having one year of life, or gambling and earning the average of the utilities for the demon’s offered lifespan and immediate death (weighted by their probabilities). If the expected utility of the gamble beats the utility of not gambling, then they accept the offer. I am going to assume that, because of the angel’s blessing, the longer they live the higher their utility.^[2] We also know that the utility of death is not negative infinity for an economist, because otherwise the negligible risk of death from everyday activities like driving and going outside still have infinite cost, implying they would cower in fear in their homes all day instead. Therefore, the demon can always offer a high enough lifespan to make the trade worthwhile, regardless of how low a probability of success. This virtually guarantees the economist will die.

We can remove even their 0.01% chance of near immortality with a different scenario. The demon offers to have the economist play a game. They will roll a six-sided die as many times as they like. If they ever roll a one, they lose and will die tonight. If they roll anything else, their lifespan increases to the point where their utility doubles. The risk is always worth it since the expected utility is higher, so the economist will always keep rolling and rolling and rolling until they die. This result is a riff on a famous paradox known as the St. Petersburg paradox, and it further shows how bad expected utilities are for making decisions.

Decision Advice from the Chronically Indecisive

I don’t know about you, but I want to be smarter about my decisions than an economist. Based on the last example, that seems like a low bar. Now that we’ve seen an example of what not to do, I’m going to try to create my own method and see if I can do any better. Note that I’m not trying to describe how people behave in practice here – an interesting study in psychology but presently irrelevant. Instead, I’m looking at what we should do if we are acting rationally and optimally, where we manage to avoid being duped by psychological biases.

The thing is, I’m probably the worst person to tell you how to make decisions. I’m so indecisive that when I walk into a store with too many colors of shirts I want to cry. But that does mean that I am really good at telling you how not to make decisions. So here are four requirements you should meet in order to be a reasonable and rational decision-maker.

Stop Spinning in Circles

If you simultaneously prefer apple pie to chocolate pie, chocolate pie to peach pie, and peach pie to apple pie, you need to seriously reevaluate your pie preferences. To avoid this, your preferences should have no circles like these, a property mathematicians call transitivity.

Don’t Shrug and Give Up

Which is bigger, a pound or a second? This question has no answer, because we are comparing things that are measured totally differently. If our preferences between outcomes are measured in different units like this, then there is no optimal answer for which decision we should make. At that point, I may as well start crying in the store. To save yourself from this fate, your preferences must be complete, meaning every possible outcome can be compared.

Try Not to Sweat the Infinitely Tiny Things

This next one is a bit more complicated to explain. Say you prefer a beach vacation to staying home, and prefer staying home to being stranded in the middle of the woods (let’s be honest, there are too many creepy crawlies out there). Now, consider how your preferences change when we add a tiny amount of uncertainty to this situation. Suppose that when driving to the beach, there is a 0.01% chance your car breaks down and you get stuck in the woods. But, if you choose to get lost in the trees, there’s a 0.01% chance you stumble your way out and discover a beach. Do you still prefer going for the beach vacation, even with the minuscule risk? Do you still prefer staying home to going in the woods, even with the chance of getting something better? If so, then you agree that changing risk by a tiny amount can’t cause large changes in your preferences, a property we call Archimedian after the Greek mathematician Archimedes.

There’s nothing special about 0.01% by the way. We could have used 1 in a million, or even less. As long as there’s a tiny change we can make where your preferences don’t flip, then your preferences are Archimedian. If you are indifferent between two choices, this doesn’t apply.

You might be skeptical, believing that in some scenarios adding any degree of risk is unacceptable. If you believe this, consider the fact that pretty much all of the situations we encounter in life always have a non-zero chance of failure, even if these chances are quite small. Maybe the company you work for goes bankrupt, so you don’t get paid. Maybe you get in a car crash driving home. Maybe you set your house on fire when cooking dinner. Heck, maybe the sun won’t rise tomorrow. After all, how can we be sure? But if you ever worked a job, drove a car, cooked a meal, or watched a sunrise then you’ve accepted this risk. The Achimedian property simply means that a tiny change in this risk doesn’t change your decision. It is not about going from certainty to uncertainty, because in reality we are never certain about outcomes (even if we are 99.99999% sure).

If you assume something is certain that is not logically necessary, then you violate a principle known as Cromwell’s Rule. What does Cromwell have to say about assuming something is certain?

I beseech you, in the bowels of Christ, think it possible that you may be mistaken.

I may or may not have included this entire tangent just so I could put that quote in this article. Best not get too distracted though, one more property to go.

Don’t Get Distracted

You are at a restaurant and am deciding between a rice and pasta dish you want to eat. You’re about to order the pasta, but then you notice there’s also a soup on the menu that looks good, which causes you to change my mind and order the rice instead. This sounds a little nonsensical, since whether or not there is a soup on the menu should not effect whether you prefer rice or pasta. To avoid this, your decisions should follow a principle called the independence of irrelevant alternatives.^[3] In general, you should make decisions where the existence of an “irrelevant alternative” (the soup) doesn’t change how you decide between two other options (rice or pasta).

You are actually going to need a stronger independence that can handle uncertainty. Suppose the restaurant has an absent-minded chef that has a 5% chance of mixing up your order and giving you a soup instead. Is it rational for you to say you would prefer rice with the absent-minded chef while preferring pasta with the grounded one? If so, why does the chance of getting an “irrelevant alternative” (the soup) change your preferences between the relevant ones? Generally, preferences that don’t flip when a constant uncertainty like this is introduced or changed is also said to satisfy independence of irrelevant alternatives. Our two definitions of this term mean different things, so to remove this ambiguity I am going to refer to this criterion simply as independence.

It might be reasonable to violate independence in some scenarios. You might get stressed out by the chance of an order mix up, and while you prefer pasta generally you find rice to be a better stress relief food. This would cause your preferences to flip due to the seemingly irrelevant chance of being given soup. This makes sense for restaurant foods where it is all up to taste anyway, but if we start to consider more serious decisions like how to triage patients at a hospital or which stocks to invest in with your life savings, it seems intuitive to me that we don’t want this psychological effect to bias the results. We shouldn’t be phased by an increased chance of failure when there’s nothing we can do about it. I feel like our demon deal should be part of the serious decisions camp as well, so I think that your preferences should probably be independent.

All Together Now

At this point, we’ve defined a particular flavor of rationality known as VNM-rationality, named after the pair who originally defined it: John von Neumann and Oskar Morgenstern. You are VNM-rational in regards to your decision making if your preferences are:

1. Transitive
2. Complete
3. Archimedean
4. Independent

Do you agree with these requirements? If so, then joke’s on you, sucker!! You’re no better than a meager economist! This is the result of the von Neumann-Morgenstern utility theorem, which states that the only way to be VNM-rational is to maximize expected utility. You might not consciously realize what utility function you’re maximizing, but it’s there regardless. This also means you get stuck with all the problems the expected utility method has, from the St. Petersburg paradox to trading your life to the demon.

Wait, Where Did We Go Wrong?

I make fun of economists a lot here, but I hope that this shows that their method isn’t as unreasonable as it first appears. A relatively intuitive sense of how to make decisions correctly basically requires we follow their methodology: put a price on everything and take averages under uncertainty. This method works best when there are repeatable chances like in stock markets, making it particularly suitable for economists. In the market, someone maximizing their expected cash will beat everyone else in the long run.^[4]

But the methods that work best in the long run aren’t always the best methods. As the famous saying goes,

In the long run, we are all dead.

– John Maynard Keynes

If we always optimize for the long-run then we never optimize for the present, and that will lead to problems. Not to mention that we need accurate probabilities to compute expected utilities in the first place, which we often can’t do because of unknown unknowns.

Furthermore, when probabilities are extremely low, rewards are extremely high, or the opportunity to take the chance is rare, we can’t rely on the long-run average smoothing out the risk. This is ultimately the problem that the demon deal and the St. Petersburg paradox demonstrated. The article this post is based on, entitled “A Paradox for Tiny Probabilities and Enormous Values,” covers why specifically these situations lead to problems, and how it extends beyond expected utility methods. The paper concluded that, even if we reject VNM-rationality, we are stuck accepting one of the following warts in any decision framework:

• Recklessness: acting like the economist with the demon deal and gambling everything away over a tiny probability of massive success.
• Timidity: being unwilling to accept a tiny increase in risk no matter how great the rewards.
• Nontransitivity: having cycles in our preferences.
• Boundedness: having a limit to our maximum utility, implying there is a point where we become apathetic about everything and nothing can make us happier.

The paper goes into more details regarding why all of these options suck. This means that I could take the time giving you all the reasons to reject one of the VNM requirements, but it wouldn’t matter much because the paper implies that a new theory with different rules is bound to have its own problems. Large numbers just break things, and we can’t do anything about it.

I think the lesson here is that rationality and decision making are hard. It’s not just about being smart or thoughtful, but often about coming up with solutions to seemingly impossible questions. It might be that rationality only exist in theory, the spherical cow of decision-making that we can’t actually achieve in reality. This doesn’t mean rationality is not worth striving for, though it does mean you’re going to have to accept some imperfections no matter how you make your decisions.

So maybe we shouldn’t be judging economists for how their theory works at its worst, and instead celebrate how useful and simple it can be at its best. Maybe my friends should stop giving me a hard time when I can’t make a decision. And maybe we should be expected utility maximizers. But I would be a bit more skeptical about deals offered by demons in your dreams.

---

Footnotes

1. Utility actually has two definitions that are subtly different. One definition is how good something is for you, which is what we are concerned with here. The other is how good something is for society at large. This requires that we have some way to aggregate individual utilities into a group utility, like summing or averaging them. Unfortunately, these aggregation methods often lead to repugnant conclusions. The idea that we should maximize this group utility is the ethical philosophy of utilitarianism. ↩︎

2. It might be tempting to say that you cannot gain any utility beyond a lifespan of 30,000 years or so. But maybe in 30,000 years your favorite great-great-…-great grandchild is getting inaugurated as President of Mars and you wish you could stick around long enough to see it. Or maybe humanity calls for your vast wisdom gained over many millenia to help with a coming crisis and you wish you had time to respond before you kicked the bucket. In both of these cases, you would have a positive utility from extending your life.

   In general, there should always be a non-zero chance that you will be in this kind of situation after any length of time where you wish to live longer. And since we are guaranteed health and happiness, increasing our lifespan should never make us worse off. These imply that expected utility will always keep growing as lifespan increases.

   Technically, we need more than just infinite growth, our growth also needs to be unbounded. We can increase our utility for forever while keeping it below a threshold, just like how 1 + 1/2 + 1/4 + 1/8 + … grows forever but never goes above 2. Whether this assumption is justified is harder to tell, but ultimately this demon deal is just a metaphor I’m using to explain a broader point so I’ll be assuming this by fiat regardless. ↩︎

3. This explanation is based on a quote on the Wikipedia page for independence of irrelevant alternatives attributed to Sidney Morgenbesser, using pies instead of dinner dishes. ↩︎

4. There are reasons to believe that the utility function for money is logarithmic due to the fact that “it takes money to make money,” implying a loss of money is also a loss of potential future earnings. This can be seen more formally in the justification for the Kelly criterion, which states that in order to maximize your wealth in the long term, you should invest to maximize the logarithm of your wealth in the short term. ↩︎