# Dialogue on Appeals to Consequences

[note: the following is essentially an expanded version of this LessWrong comment on whether appeals to consequences are normative in discourse. I am exasperated that this is even up for debate, but I figure that making the argumentation here explicit is helpful]

Carter and Quinn are discussing charitable matters in the town square, with a few onlookers.

Carter: “So, this local charity, People Against Drowning Puppies (PADP), is nominally opposed to drowning puppies.”

Quinn: “Of course.”

Carter: “And they said they’d saved 2170 puppies last year, whereas their total spending was $1.2 million, so they estimate they save one puppy per$553.”

Carter: “So, I actually checked with some of their former employees, and if what they say and my corresponding calculations are right, they actually only saved 138 puppies.”

Quinn: “Hold it right there. Regardless of whether that’s true, it’s bad to say that.”

Carter: “That’s an appeal to consequences, well-known to be a logical fallacy.”

Quinn: “Is that really a fallacy, though? If saying something has bad consequences, isn’t it normative not to say it?”

Carter: “Well, for my own personal decisionmaking, I’m broadly a consequentialist, so, yes.”

Quinn: “Well, it follows that appeals to consequences are valid.”

Carter: “It isn’t logically valid. If saying something has bad consequences, that doesn’t make it false.”

Quinn: “But it is decision-theoretically compelling, right?”

Carter: “In theory, if it could be proven, yes. But, you haven’t offered any proof, just a statement that it’s bad.”

Quinn: “Okay, let’s discuss that. My argument is: PADP is a good charity. Therefore, they should be getting more donations. Saying that they didn’t save as many puppies as they claimed they did, in public (as you just did), is going to result in them getting fewer donations. Therefore, your saying that they didn’t save as many puppies as they claimed to is bad, and is causing more puppies to drown.”

Carter: “While I could spend more effort to refute that argument, I’ll initially note that you only took into account a single effect (people donating less to PADP) and neglected other effects (such as people having more accurate beliefs about how charities work).”

Quinn: “Still, you have to admit that my case is plausible, and that some onlookers are convinced.”

Carter: “Yes, it’s plausible, in that I don’t have a full refutation, and my models have a lot of uncertainty. This gets into some complicated decision theory and sociological modeling. I’m afraid we’ve gotten sidetracked from the relatively clear conversation, about how many puppies PADP saved, to a relatively unclear one, about the decision theory of making actual charity effectiveness clear to the public.”

Quinn: “Well, sure, we’re into the weeds now, but this is important! If it’s actually bad to say what you said, it’s important that this is widely recognized, so that we can have fewer… mistakes like that.”

Carter: “That’s correct, but I feel like I might be getting trolled. Anyway, I think you’re shooting the messenger: when I started criticizing PADP, you turned around and made the criticism about me saying that, directing attention against PADP’s possible fraudulent activity.”

Quinn: “You still haven’t refuted my argument. If you don’t do so, I win by default.”

Carter: “I’d really rather that we just outlaw appeals to consequences, but, fine, as long as we’re here, I’m going to do this, and it’ll be a learning experience for everyone involved. First, you said that PADP is a good charity. Why do you think this?”

Quinn: “Well, I know the people there and they seem nice and hardworking.”

Carter: “But, they said they saved over 2000 puppies last year, when they actually only saved 138, indicating some important dishonesty and ineffectiveness going on.”

Carter: “Hold up! We’re in the middle of evaluating your argument that saying that is bad! You can’t use the conclusion of this argument in the course of proving it! That’s circular reasoning!”

Quinn: “Fine. Let’s try something else. You said they’re being dishonest. But, I know them, and they wouldn’t tell a lie, consciously, although it’s possible that they might have some motivated reasoning, which is totally different. It’s really uncivil to call them dishonest like that. If everyone did that with the willingness you had to do so, that would lead to an all-out rhetorical war…”

Carter: “God damn it. You’re making another appeal to consequences.”

Quinn: “Yes, because I think appeals to consequences are normative.”

Carter: “Look, at the start of this conversation, your argument was that saying PADP only saved 138 puppies is bad.”

Quinn: “Yes.”

Carter: “And now you’re in the course of arguing that it’s bad.”

Quinn: “Yes.”

Carter: “Whether it’s bad is a matter of fact.”

Quinn: “Yes.”

Carter: “So we have to be trying to get the right answer, when we’re determining whether it’s bad.”

Quinn: “Yes.”

Carter: “And, while appeals to consequences may be decision theoretically compelling, they don’t directly bear on the facts.”

Quinn: “Yes.”

Carter: “So we shouldn’t have appeals to consequences in conversations about whether the consequences of saying something is bad.”

Quinn: “Why not?”

Carter: “Because we’re trying to get to the truth.”

Quinn: “But aren’t we also trying to avoid all-out rhetorical wars, and puppies drowning?”

Carter: “If we want to do those things, we have to do them by getting to the truth.”

Quinn: “The truth, according to your opinion-

Carter: “God damn it, you just keep trolling me, so we never get to discuss the actual facts. God damn it. Fuck you.”

Quinn: “Now you’re just spouting insults. That’s really irresponsible, given that I just accused you of doing something bad, and causing more puppies to drown.”

Carter: “You just keep controlling the conversation by OODA looping faster than me, though. I can’t refute your argument, because you appeal to consequences again in the middle of the refutation. And then we go another step down the ladder, and never get to the truth.”

Quinn: “So what do you expect me to do? Let you insult well-reputed animal welfare workers by calling them dishonest?”

Carter: “Yes! I’m modeling the PADP situation using decision-theoretic models, which require me to represent the knowledge states and optimization pressures exerted by different agents (both conscious and unconscious), including when these optimization pressures are towards deception, and even when this deception is unconscious!”

Quinn: “Sounds like a bunch of nerd talk. Can you speak more plainly?”

Carter: “I’m modeling the actual facts of how PADP operates and how effective they are, not just how well-liked the people are.”

Quinn: “Wow, that’s a strawman.”

Carter: “Look, how do you think arguments are supposed to work, exactly? Whoever is best at claiming that their opponent’s argumentation is evil wins?”

Quinn: “Sure, isn’t that the same thing as who’s making better arguments?”

Carter: “If we argue by proving our statements are true, we reach the truth, and thereby reach the good. If we argue by proving each other are being evil, we don’t reach the truth, nor the good.”

Quinn: “In this case, though, we’re talking about drowning puppies. Surely, the good in this case is causing fewer puppies to drown, and directing more resources to the people saving them.”

Carter: “That’s under contention, though! If PADP is lying about how many puppies they’re saving, they’re making the epistemology of the puppy-saving field worse, leading to fewer puppies being saved. And, they’re taking money away from the next-best-looking charity, which is probably more effective if, unlike PADP, they’re not lying.”

Quinn: “How do you know that, though? How do you know the money wouldn’t go to things other than saving drowning puppies if it weren’t for PADP?”

Carter: “I don’t know that. My guess is that the money might go to other animal welfare charities that claim high cost-effectiveness.”

# Security and boundaries

I’ve already kind of answered this question by saying that ensuring that material goods can be used in the future is a security problem. If you use one of your material goods to produce another material good, and someone takes this new good, then you can’t put this good back into your production process. Thus, what would have been a positive feedback loop is instead a negative feedback loop, as it leaks goods faster than it produces them.

Solving security issues generally requires boundaries. You need to draw a boundary in material space somewhere, differentiating the inside from the outside, such that material goods (such as energy) on the inside don’t leak out, and can potentially have positive feedback loops. There are many ways to prevent leaks across a boundary while still allowing informational and material to pass through sometimes, such as semiporous physical barriers and active policing. Regardless of the method to enforce the boundary, the boundary has to exist in some geometrical sense for it to make sense to say that e.g. energy increases within this system.

Not all security issues are from other agents; some are from non-agentic processes. Consider a homeostatic animal. If the animal expends energy to warm its body, and this warmth escapes, the animal will fail to realize gains from the energy expenditure. Thus, the animal has a boundary (namely, skin) to solve this “security problem”. The cold air particles that take away heat from the animal are analogous to agents that directly take resources, though obviously less agentic. While perhaps my usage of the word “security” to include responses to nonagentic threats is nonstandard, I hope it is clear that these are on the same spectrum as agentic threats, and can be dealt with in some of the same ways.

It is also worth thinking about semi-agentic entities, such as microorganisms. One of the biggest threats to a food store is microorganisms (i.e. rotting), and slowing the negative feedback loops depleting food stores requires solving this security problem using a boundary (such as a sealed container or a subset of the air that is colder than the outside air, such as in a refrigerator).

Property rights are a simple example of boundaries. Certain goods are considered to be “owned” by different parties, such that there is common agreement about who owns what, and people are for one reason or another not motivated to take other people’s stuff. Such division of goods into sets owned by different parties is a set of boundaries enabling positive feedback loops, which are especially salient in capitalism.

What about trust between different entities? A complex ecosystem will contain entities satisfying a variety of niches, which include parasitism and predation (which are on the same spectrum). A trust network can be thought of as a way for different entities to draw various boundaries, often fuzzy ones, that mostly exclude parasites/predators, such that there are few leaks from inside this boundary to outside this boundary (which would include parasitism/predation by entities outside the boundary). There are “those who you trust” and “those who you don’t trust” (both fuzzy sets), and you assign more utility to giving resources to those you trust, as this allows for positive feedback loops within a system that contains you (namely, the trust network).

# Externalities and sustainability

Since no subsystem of the world is causally closed, all positive feedback loops have externalities. By definition, the outside world is only directly affected by these externalities, and is only affected by what happens within the boundary to the extent that this eventually leads to externalities. A wise designer of a positive feedback loop will anticipate its externalities, and set it up such that the externalities are overall desirable to the designer. After all, there is no point to creating a positive feedback loop unless its externalities are mostly positive.

A positive feedback loop’s externalities modify its environment, affecting its own ability to continue; for example, a positive feedback loop of microorganisms eating food will exhaust itself by consuming the food. So, different positive feedback loops are environmentally sustainable to different extents. Both production and conquest generate positive feedback loops, as Ben Hoffman discusses in this post, but production is much more environmentally sustainable than conquest.

One way to increase environmental sustainability is to move more processes to the inside of the boundary. For example, a country that is consuming large amounts of iron (driving up iron prices) may consider setting up its own iron mines. Thus, the inside of the boundary becomes more like an economy of its own. This is sometimes known as import replacement.

Of course, the environmental sustainability of a positive feedback loop can also be a negative, as it is better for some processes (such as rotting) to limit or exhaust themselves, thus transitioning to negative feedback or a combination of positive and negative feedback. Processes that include intentionally-designed positive and negative feedback can be much more environmentally sustainable than processes that only have positive feedback loops designed in, since they can limit their growth when such growth would be unsustainable.

While in theory the philosophy of effective altruism (EA) would imply a strong (and likely overwhelming) emphasis on creating and maintaining environmentally sustainable positive feedback loops with positive externalities, typically-recommended EA practices (such as giving away 10% of one’s income) are negative feedback loops (the more you make, the more you give away). While in theory the place the resources are given to could have a faster positive feedback loop than just investing in yourself, your friends, and your projects, in practice I rarely believe claims of this form that come from the EA movement; for example, if a country has a high rate of poverty, that indicates that the negative feedback loops (such as corruption) are likely stronger than the positive ones, and that giving resources is ineffective. Thus, I cannot in good conscience allow anything like current EA ideology to substantially control resource allocation in most systems I create, even though EA philosophy taken to its logical conclusion would get the right answer on the importance of securing the boundaries of positive feedback loops.

# Policy suggestions

How do these ideas translate to action? One suggestion is that, if you are trying to do something big, you use one or more positive feedback loops, and ask yourself the following questions about each one:

1. What’s the generator of my positive feedback loop (i.e. what’s the process that turns stuff into more stuff)?
2. What is the boundary within which the positive feedback increases resources?
3. How am I reducing leakage across this boundary?
4. What are the externalities of this positive feedback loop?
5. How environmentally sustainable is this positive feedback loop?
6. Are there built-in negative feedback loops that increase environmental sustainability?

(thanks to Bryce Hidysmith for a conversation that led to this post)

# Act of Charity

(Also posted on LessWrong)

The stories and information posted here are artistic works of fiction and falsehood. Only a fool would take anything posted here as fact.

—Anonymous

# Act I.

Carl walked through the downtown. He came across a charity stall. The charity worker at the stall called out, “Food for the Africans. Helps with local autonomy and environmental sustainability. Have a heart and help them out.” Carl glanced at the stall’s poster. Along with pictures of emaciated children, it displayed infographics about how global warming would cause problems for African communities’ food production, and numbers about how easy it is to help out with money. But something caught Carl’s eye. In the top left, in bold font, the poster read, “IT IS ALL AN ACT. ASK FOR DETAILS.”

Carl: “It’s all an act, huh? What do you mean?”

Worker: “All of it. This charity stall. The information on the poster. The charity itself. All the other charities like us. The whole Western idea of charity, really.”

Carl: “Care to clarify?”

Worker: “Sure. This poster contains some correct information. But a lot of it is presented in a misleading fashion, and a lot of it is just lies. We designed the poster this way because it fits with people’s idea is of a good charity they should give money to. It’s a prop in the act.”

Carl: “Wait, the stuff about global warming and food production is a lie?”

Worker: “No, that part is actually true. But in context we’re presenting it as some kind of imminent crisis that requires an immediate infusion of resources, when really it’s a very long-term problem that will require gradual adjustment of agricultural techniques, locations, and policies.”

Carl: “Okay, that doesn’t actually sound like more of a lie than most charities tell.”

Worker: “Exactly! It’s all an act.”

Carl: “So why don’t you tell the truth anyway?”

Worker: “Like I said before, we’re trying to fit with people’s idea of what a charity they should give money to looks like. More to the point, we want them to feel compelled to give us money. And they are compelled by some acts, but not by others. The idea of an immediate food crisis creates more moral and social pressure towards immediate action, than the idea that there will be long-term agricultural problems that require adjustments.

Carl: “That sounds…kind of scammy?”

Worker: “Yes, you’re starting to get it! The act is about violence! It’s all violence!”

Carl: “Now hold on, that seems like a false equivalence. Even if they were scammed by you, they still gave you money of their own free will.”

Worker: “Most people, at some level, know we’re lying to them. Their eyes glaze over ‘IT IS ALL AN ACT’ as if it were just a regulatory requirement to put this on charity posters. So why would they give money to a charity that lies to them? Why do you think?”

Carl: “I’m not nearly as sure as you that they know this! Anyway, even if they know at some level it’s a lie, that doesn’t mean they consciously know, so to their conscious mind it seems like being completely heartless.”

Worker: “Exactly, it’s emotional blackmail. I even say ‘Have a heart and help them out’. So if they don’t give us money, there’s a really convenient story that says they’re heartless, and a lot of them will even start thinking about themselves that way. Having that story told about them opens them up to violence.”

Carl: “How?”

Worker: “Remember Martin Shkreli?”

Carl: “Yeah, that asshole who jacked up the Daraprim prices.”

Worker: “Right. He ended up going to prison. Nominally, it was for securities fraud. But it’s not actually clear that whatever security fraud he did was worse than what others in his industry were doing. Rather, it seems likely that he was especially targeted because he was a heartless asshole.”

Carl: “But he still broke the law!”

Worker: “How long would you be in jail if you got punished for every time you had broken the law?”

Carl: “Well, I’ve done a few different types of illegal drugs, so… a lot of years.”

Worker: “Exactly. Almost everyone is breaking the law. So it’s really, really easy for the law to be enforced selectively, to punish just about anyone. And the people who get punished the most are those who are villains in the act.”

Carl: “Hold on. I don’t think someone would actually get sent to prison because they didn’t give you money.”

Worker: “Yeah, that’s pretty unlikely. But things like it will happen. People are more likely to give if they’re walking with other people. I infer that they believe they will be abandoned if they do not give.”

Carl: “That’s a far cry from violence.”

Worker: “Think about the context. When you were a baby, you relied on your parents to provide for you, and abandonment by them would have meant certain death. In the environment of evolutionary adaptation, being abandoned by your band would have been close to a death sentence. This isn’t true in the modern world, but people’s brains mostly don’t really distinguish abandonment from violence, and we exploit that.”

Carl: “That makes some sense. I still object to calling it violence, if only because we need a consistent definition of ‘violence’ to coordinate, well, violence against those that are violent. Anyway, I get that this poster is an act, and the things you say to people walking down the street are an act, but what about the charity itself? Do you actually do the things you say you do?”

Worker: “Well, kind of. We actually do give these people cows and stuff, like the poster says. But that isn’t our main focus, and the main reason we do it is, again, because of the act.”

Carl: “Because of the act? Don’t you care about these people?”

Worker: “Kind of. I mean, I do care about them, but I care about myself and my friends more; that’s just how humans work. And if it doesn’t cost me much, I will help them. But I won’t help them if it puts our charity in a significantly worse position.”

Carl: “So you’re the heartless one.”

Worker: “Yes, and so is everyone else. Because the standard you’re set for ‘not heartless’ is not one that any human actually achieves. They just deceive themselves about how much they care about random strangers; the part of their brain that inserts these self-deceptions into their conscious narratives is definitely not especially altruistic!”

Carl: “According to your own poster, there’s going to be famine, though! Is the famine all an act to you?”

Worker: “No! Famine isn’t an act, but most of our activities in relation to it are. We give people cows because that’s one of the standard things charities like ours are supposed to do, and it looks like we’re giving these people local autonomy and stuff.”

Carl: “Looks like? So this is all just optics?”

Worker: “Yes! Exactly!”

Carl: “I’m actually really angry right now. You are a terrible person, and your charity is terrible, and you should die in a fire.”

Worker: “Hey, let’s actually think through this ethical question together. There’s a charity pretty similar to ours that’s set up a stall a couple blocks from here. Have you seen it?”

Carl: “Yes. They do something with water filtering in Africa.”

Worker: “Well, do you think their poster is more or less accurate than ours?”

Carl: “Well, I know yours is a lie, so…”

Worker: “Hold on. This is Gell-Mann amnesia. You know ours is a lie because I told you. This should adjust your model of how charities work in general.”

Carl: “Well, it’s still plausible that they are effective, so I can’t condemn—”

Worker: “Stop. In talking of plausibility rather than probability, you are uncritically participating in the act. You are taking symbols at face value, unless there is clear disproof of them. So you will act like you believe any claim that’s ‘plausible’, in other words one that can’t be disproven from within the act. You have never, at any point, checked whether either charity is doing anything in the actual, material world.”

Carl: “…I suppose so. What’s your point, anyway?”

Worker: “You’re shooting the messenger. All or nearly all of these charities are scams. Believe me, we’ve spent time visiting these other organizations, and they’re universally fraudulent, they just have less self-awareness about it. You’re only morally outraged at the ones that don’t hide it. So your moral outrage optimizes against your own information. By being morally outraged at us, you are asking to be lied to.”

Carl: “Way to blame the victim. You’re the one lying.”

Worker: “We’re part of the same ecosystem. By rewarding a behavior, you cause more of it. By punishing it, you cause less of it. You reward lies that have plausible deniability and punish truth, when that truth is told by sinners. You’re actively encouraging more of the thing that is destroying your own information!”

Carl: “It still seems pretty strange to think that they’re all scams. Like, some of my classmates from college went into the charity sector. And giving cows to people who have food problems actually seems pretty reasonable.”

Worker: “It’s well known by development economists that aid generally creates dependence, that in giving cows to people we disrupt their local economy’s cow market, reducing the incentive to raise cattle. And in theory it could still be worth it, but our preliminary calculations indicate that it probably isn’t.”

Carl: “Hold on. You actually ran the calculation, found that your intervention was net harmful, and then kept doing it?”

Worker: “Yes. Again, it is all—”

Carl: “What the fuck, seriously? You’re a terrible person.”

Worker: “Do you think any charity other than us would have run the calculation we did, and then actually believe the result? Or would they have fudged the numbers here and there, and when even a calculation with fudged numbers indicated that the intervention was ineffective, come up with a reason to discredit this calculation and replace it with a different one that got the result they wanted?”

Carl: “Maybe a few… but I see your point. But there’s a big difference between acting immorally because you deceived yourself, and acting immorally with a clear picture of what you’re doing.”

Worker: “Yes, the second one is much less bad!”

Carl: “What?”

Worker: “All else being equal, it’s better to have clearer beliefs than muddier ones, right?”

Carl: “Yes. But in this case, it’s very clear that the person with the clear picture is acting immorally, while the self-deceiver, uhh..”

Worker: “…has plausible deniability. Their stories are plausible even though they are false, so they have more privilege within the act. They gain privilege by muddying the waters, or in other words, destroying information.”

Carl: “Wait, are you saying self-deception is a choice?”

Worker: “Yes! It’s called ‘motivated cognition’ for a reason. Your brain runs something like a utility-maximization algorithm to tell when and how you should deceive yourself. It’s epistemically correct to take the intentional stance towards this process.”

Carl: “But I don’t have any control over this process!”

Worker: “Not consciously, no. But you can notice the situation you’re in, think about what pressures there are on you to self-deceive, and think about modifying your situation to reduce these pressures. And you can do this to other people, too.”

Carl: “Are you saying everyone is morally obligated to do this?”

Worker: “No, but it might be in your interest, since it increases your capabilities.”

Carl: “Why don’t you just run a more effective charity, and advertise on that? Then you can outcompete the other charities.”

Worker: “That’s not fashionable anymore. The ‘effectiveness’ branding has been tried before; donors are tired of it by now. Perhaps this is partially because there aren’t functional systems that actually check which organizations are effective and which aren’t, so scam charities branding themselves as effective end up outcompeting the actually effective ones. And there are organizations claiming to evaluate charities’ effectiveness, but they’ve largely also become scams by now, for exactly the same reasons. The fashionable branding now is environmentalism.”

Carl: “This is completely disgusting. Fashion doesn’t help people. Your entire sector is morally depraved.”

Worker: “You are entirely correct to be disgusted. This moral depravity is a result of dysfunctional institutions. You can see it outside charity too; schools are authoritarian prisons that don’t even help students learn, courts put people in cages for not spending enough on a lawyer, the US military blows up civilians unnecessarily, and so on. But you already knew all that, and ranting about these things is itself a trope. It is difficult to talk about how broken the systems are without this talking itself being interpreted as merely a cynical act. That’s how deep this goes. Please actually update on this rather than having your eyes glaze over!”

Carl: “How do you even deal with this?”

Worker: “It’s already the reality you’ve lived in your whole life. The only adjustment is to realize it, and be able to talk about it, without this destroying your ability to participate in the act when it’s necessary to do so. Maybe functional information-processing institutions will be built someday, but we are stuck with this situation for now, and we’ll have no hope of building functional institutions if we don’t understand our current situation.”

Carl: “You are wasting so much potential! With your ability to see social reality, you could be doing all kinds of things! If everyone who were as insightful as you were as pathetically lazy as you, there would be no way out of this mess!”

Worker: “Yeah, you’re right about that, and I might do something more ambitious someday, but I don’t really want to right now. So here I am. Anyway… food for the Africans. Helps with local autonomy and environmental sustainability. Have a heart and help them out.”

Carl sighed, fished a 10 dollar bill from his wallet, and gave it to the charity worker.

# Decision theory and zero-sum game theory, NP and PSPACE

(Also posted on LessWrong)

At a rough level:

• Decision theory is about making decisions to maximize some objective function.
• Zero-sum game theory is about making decisions to optimize some objective function while someone else is making decisions to minimize this objective function.

These are quite different.

## Decision theory and NP

Decision theory roughly corresponds to the NP complexity class.  Consider the following problem:

Given a set of items, each of which has a integer-valued value and weight, does there exist a subset with total weight less than $w$ and total value at least $v$?

(It turns out that finding a solution is not much harder than determining whether there is a solution; if you know how to tell whether there is a solution to arbitrary problems of this form, you can in particular tell if there is a solution that uses any particular item.)

This is the knapsack problem, and it is in NP.  Given a candidate solution, it is easy to check whether it actually is a solution: you just count the values and the weights.  Since this solution would constitute a proof that the answer to the question is “yes”, and a solution exists whenever the answer is “yes”, this problem is in NP.

The following is a general form for NP problems:

$\exists x_1 \in \{0, 1\} \exists x_2 \in \{0, 1\} \ldots \exists x_k \in \{0, 1\} f(x_1, ..., x_k)$

where $f$ is a specification of a circuit (say, made of AND, OR, and NOT gates) that outputs a single Boolean value.  That is, the problem is to decide whether there is some assignment of values to $x_1, \ldots, x_k$ that $f$ outputs true on.  This is a variant of the Boolean satisfiability problem.

In decision theory (and in NP), all optimization is in the same direction.  The only quantifier is $\exists$.

## Zero-sum game theory and PSPACE

Zero-sum game theory roughly corresponds to the PSPACE complexity class.  Consider the following problem:

Given a specification of a Reversi game state (on an arbitrarily-large square board), does there exists a policy for the light player that guarantees a win?

(It turns out that winning the game is not much harder than determining whether there is a winning policy; if you know how to tell whether there is a solution to arbitrary problems of this form, then in particular you can tell if dark can win given a starting move by light.)

This problem is in PSPACE: it can be solved by a Turing machine using a polynomial amount of space.  This Turing machine works through the minimax algorithm: it simulates all possible games in a backtracking fashion.

The following is a general form for PSPACE problems:

$\exists x_1 \in \{0, 1\} \forall y_1 \in \{0, 1\} \ldots \exists x_k \in \{0, 1\} \forall y_k \in \{0, 1\} f(x_1, y_1, \ldots, x_k, y_k)$

where $f$ is a specification of a circuit (say, made of AND, OR, and NOT gates) that outputs a single Boolean value.  That is, the problem is to determine whether it is possible to set the $x$ values interleaved with an opponent setting the $y$ values such that, no matter how the opponent acts, $f(x_1, y_1, \ldots, x_k, y_k)$ is true.  This is a variant of the quantified Boolean formula problem.  (Interpreting a logical formula containing $\exists$ and $\forall$ as a game is standard; see game semantics).

In zero-sum game theory, all optimization is in one of two completely opposite directions.  There is literally no difference between something that is good for one player and something that is bad for the other.  The opposing quantifiers $\exists$ and $\forall$, representing decisions by the two opponents, are interleaved.

## Different cognitive modes

The comparison to complexity classes suggests that there are two different cognitive modes for decision theory and zero-sum game theory, as there are two different types of algorithms for NP-like and PSPACE-like problems.

In decision theory, you plan with no regard to any opponents interfering with your plans, allowing you to plan on arbitrarily long time scales.  In zero-sum game theory, you plan on the assumption that your opponent will interfere with your plans (your $\exists$s are interleaved with your opponent’s $\forall$s), so you can only plan as far as your opponent lacks the ability to interfere with these plans.  You must have a short OODA loop, or your opponent’s interference will make your plans useless.

In decision theory, you can mostly run on naïve expected utility analysis: just do things that seem like they will work.  In zero-sum game theory, you must screen your plans for defensibility: they must be resistant to possible attacks.  Compare farming with border defense, mechanical engineering with computer security.

High-reliability engineering is an intermediate case: designs must be selected to work with high probability across a variety of conditions, but there is normally no intelligent optimization power working against the design.  One could think of nature as an “adversary” selecting some condition to test the design against, and represent this selection by a universal quantifier; however, this is qualitatively different from a true adversary, who applies intentional optimization to break a design rather than haphazard selection of conditions.

## Conclusion

These two types of problems do not cover all realistic situations an agent might face.  Decision problems involving agents with different but not completely opposed objective functions are different, as are zero-sum games with more than two players.  But realistic situations share some properties with each of these, and I suspect that there might actually be a discrete distinction between cognitive modes for NP-like decision theory problems and PSPACE-like zero-sum games.

What’s the upshot?  If you want to know what is going on, one of the most important questions (perhaps the most important question) is: what kind of game are you playing?  Is your situation more like a decision theory problem or a zero-sum game?  To what extent is optimization by different agents going in the same direction, opposing directions, or orthogonal directions?  What would have to change for the nature of the game to change?

Thanks to Michael Vassar for drawing my attention to the distinction between decision theory and zero-sum game theory as a distinction between two cognitive modes.

Related: The Face of the Ice