Boundaries enable positive material-informational feedback loops

(Also posted on LessWrong)

[epistemic status: obvious once considered, I think]

If you want to get big things done, you almost certainly need positive feedback loops. Unless you can already do all the necessary things, you need to do/make things that allow you to do/make more things in the future. This dynamic can be found in RPG and economy-management games, and in some actual economic systems, such as industrializing economies.

Material, information, and economy

Some goods that can be used in a positive feedback loop, such as software and inventions, are informational. Once produced, they can be used indefinitely in the future. In economic terms, they are nonrivalous.

Other goods are material, such as manufactured goods and energy. They can’t be copied cheaply. In economic terms, they are rivalrous.

In practice, any long-lasting positive feedback loop contains both informational and material goods, as production of information requires a physical substrate. While ensuring that informational goods can be used in the future is an organization and communication problem (a subject beyond the scope of this post), the problem of ensuring that material goods can be used in the future is additionally a security problem.

An important question to ask is: why haven’t material-informational positive feedback loops already taken over the world? Why don’t we have so much stuff by now that providing for people’s material needs (such as food and housing) is trivial?

To some extent, material-informational positive feedback loops have taken over the world, but they seem much slower than one would naively expect. See cost disease. As an example of cost disease, the average cost of a new house in the USA has quadrupled over a 60-year period (adjusted for inflation!), whereas models of capitalism based on economy-management games such as Factorio (or, more academically, according to the labor/capital based economic models of classical economists such as David Ricardo) would suggest that houses would be plentiful by now. (And no, this isn’t just because of land prices; it costs about $300K to build a house in the US 2018)

Security and boundaries

I’ve already kind of answered this question by saying that ensuring that material goods can be used in the future is a security problem. If you use one of your material goods to produce another material good, and someone takes this new good, then you can’t put this good back into your production process. Thus, what would have been a positive feedback loop is instead a negative feedback loop, as it leaks goods faster than it produces them.

Solving security issues generally requires boundaries. You need to draw a boundary in material space somewhere, differentiating the inside from the outside, such that material goods (such as energy) on the inside don’t leak out, and can potentially have positive feedback loops. There are many ways to prevent leaks across a boundary while still allowing informational and material to pass through sometimes, such as semiporous physical barriers and active policing. Regardless of the method to enforce the boundary, the boundary has to exist in some geometrical sense for it to make sense to say that e.g. energy increases within this system.

Not all security issues are from other agents; some are from non-agentic processes. Consider a homeostatic animal. If the animal expends energy to warm its body, and this warmth escapes, the animal will fail to realize gains from the energy expenditure. Thus, the animal has a boundary (namely, skin) to solve this “security problem”. The cold air particles that take away heat from the animal are analogous to agents that directly take resources, though obviously less agentic. While perhaps my usage of the word “security” to include responses to nonagentic threats is nonstandard, I hope it is clear that these are on the same spectrum as agentic threats, and can be dealt with in some of the same ways.

It is also worth thinking about semi-agentic entities, such as microorganisms. One of the biggest threats to a food store is microorganisms (i.e. rotting), and slowing the negative feedback loops depleting food stores requires solving this security problem using a boundary (such as a sealed container or a subset of the air that is colder than the outside air, such as in a refrigerator).

Property rights are a simple example of boundaries. Certain goods are considered to be “owned” by different parties, such that there is common agreement about who owns what, and people are for one reason or another not motivated to take other people’s stuff. Such division of goods into sets owned by different parties is a set of boundaries enabling positive feedback loops, which are especially salient in capitalism.

What about trust between different entities? A complex ecosystem will contain entities satisfying a variety of niches, which include parasitism and predation (which are on the same spectrum). A trust network can be thought of as a way for different entities to draw various boundaries, often fuzzy ones, that mostly exclude parasites/predators, such that there are few leaks from inside this boundary to outside this boundary (which would include parasitism/predation by entities outside the boundary). There are “those who you trust” and “those who you don’t trust” (both fuzzy sets), and you assign more utility to giving resources to those you trust, as this allows for positive feedback loops within a system that contains you (namely, the trust network).

Externalities and sustainability

Since no subsystem of the world is causally closed, all positive feedback loops have externalities. By definition, the outside world is only directly affected by these externalities, and is only affected by what happens within the boundary to the extent that this eventually leads to externalities. A wise designer of a positive feedback loop will anticipate its externalities, and set it up such that the externalities are overall desirable to the designer. After all, there is no point to creating a positive feedback loop unless its externalities are mostly positive.

A positive feedback loop’s externalities modify its environment, affecting its own ability to continue; for example, a positive feedback loop of microorganisms eating food will exhaust itself by consuming the food. So, different positive feedback loops are environmentally sustainable to different extents. Both production and conquest generate positive feedback loops, as Ben Hoffman discusses in this post, but production is much more environmentally sustainable than conquest.

One way to increase environmental sustainability is to move more processes to the inside of the boundary. For example, a country that is consuming large amounts of iron (driving up iron prices) may consider setting up its own iron mines. Thus, the inside of the boundary becomes more like an economy of its own. This is sometimes known as import replacement.

Of course, the environmental sustainability of a positive feedback loop can also be a negative, as it is better for some processes (such as rotting) to limit or exhaust themselves, thus transitioning to negative feedback or a combination of positive and negative feedback. Processes that include intentionally-designed positive and negative feedback can be much more environmentally sustainable than processes that only have positive feedback loops designed in, since they can limit their growth when such growth would be unsustainable.

While in theory the philosophy of effective altruism (EA) would imply a strong (and likely overwhelming) emphasis on creating and maintaining environmentally sustainable positive feedback loops with positive externalities, typically-recommended EA practices (such as giving away 10% of one’s income) are negative feedback loops (the more you make, the more you give away). While in theory the place the resources are given to could have a faster positive feedback loop than just investing in yourself, your friends, and your projects, in practice I rarely believe claims of this form that come from the EA movement; for example, if a country has a high rate of poverty, that indicates that the negative feedback loops (such as corruption) are likely stronger than the positive ones, and that giving resources is ineffective. Thus, I cannot in good conscience allow anything like current EA ideology to substantially control resource allocation in most systems I create, even though EA philosophy taken to its logical conclusion would get the right answer on the importance of securing the boundaries of positive feedback loops.

Policy suggestions

How do these ideas translate to action? One suggestion is that, if you are trying to do something big, you use one or more positive feedback loops, and ask yourself the following questions about each one:

  1. What’s the generator of my positive feedback loop (i.e. what’s the process that turns stuff into more stuff)?
  2. What is the boundary within which the positive feedback increases resources?
  3. How am I reducing leakage across this boundary?
  4. What are the externalities of this positive feedback loop?
  5. How environmentally sustainable is this positive feedback loop?
  6. Are there built-in negative feedback loops that increase environmental sustainability?

(thanks to Bryce Hidysmith for a conversation that led to this post)

Act of Charity

(Also posted on LessWrong)

The stories and information posted here are artistic works of fiction and falsehood. Only a fool would take anything posted here as fact.


Act I.

Carl walked through the downtown. He came across a charity stall. The charity worker at the stall called out, “Food for the Africans. Helps with local autonomy and environmental sustainability. Have a heart and help them out.” Carl glanced at the stall’s poster. Along with pictures of emaciated children, it displayed infographics about how global warming would cause problems for African communities’ food production, and numbers about how easy it is to help out with money. But something caught Carl’s eye. In the top left, in bold font, the poster read, “IT IS ALL AN ACT. ASK FOR DETAILS.”

Carl: “It’s all an act, huh? What do you mean?”

Worker: “All of it. This charity stall. The information on the poster. The charity itself. All the other charities like us. The whole Western idea of charity, really.”

Carl: “Care to clarify?”

Worker: “Sure. This poster contains some correct information. But a lot of it is presented in a misleading fashion, and a lot of it is just lies. We designed the poster this way because it fits with people’s idea is of a good charity they should give money to. It’s a prop in the act.”

Carl: “Wait, the stuff about global warming and food production is a lie?”

Worker: “No, that part is actually true. But in context we’re presenting it as some kind of imminent crisis that requires an immediate infusion of resources, when really it’s a very long-term problem that will require gradual adjustment of agricultural techniques, locations, and policies.”

Carl: “Okay, that doesn’t actually sound like more of a lie than most charities tell.”

Worker: “Exactly! It’s all an act.”

Carl: “So why don’t you tell the truth anyway?”

Worker: “Like I said before, we’re trying to fit with people’s idea of what a charity they should give money to looks like. More to the point, we want them to feel compelled to give us money. And they are compelled by some acts, but not by others. The idea of an immediate food crisis creates more moral and social pressure towards immediate action, than the idea that there will be long-term agricultural problems that require adjustments.

Carl: “That sounds…kind of scammy?”

Worker: “Yes, you’re starting to get it! The act is about violence! It’s all violence!”

Carl: “Now hold on, that seems like a false equivalence. Even if they were scammed by you, they still gave you money of their own free will.”

Worker: “Most people, at some level, know we’re lying to them. Their eyes glaze over ‘IT IS ALL AN ACT’ as if it were just a regulatory requirement to put this on charity posters. So why would they give money to a charity that lies to them? Why do you think?”

Carl: “I’m not nearly as sure as you that they know this! Anyway, even if they know at some level it’s a lie, that doesn’t mean they consciously know, so to their conscious mind it seems like being completely heartless.”

Worker: “Exactly, it’s emotional blackmail. I even say ‘Have a heart and help them out’. So if they don’t give us money, there’s a really convenient story that says they’re heartless, and a lot of them will even start thinking about themselves that way. Having that story told about them opens them up to violence.”

Carl: “How?”

Worker: “Remember Martin Shkreli?”

Carl: “Yeah, that asshole who jacked up the Daraprim prices.”

Worker: “Right. He ended up going to prison. Nominally, it was for securities fraud. But it’s not actually clear that whatever security fraud he did was worse than what others in his industry were doing. Rather, it seems likely that he was especially targeted because he was a heartless asshole.”

Carl: “But he still broke the law!”

Worker: “How long would you be in jail if you got punished for every time you had broken the law?”

Carl: “Well, I’ve done a few different types of illegal drugs, so… a lot of years.”

Worker: “Exactly. Almost everyone is breaking the law. So it’s really, really easy for the law to be enforced selectively, to punish just about anyone. And the people who get punished the most are those who are villains in the act.”

Carl: “Hold on. I don’t think someone would actually get sent to prison because they didn’t give you money.”

Worker: “Yeah, that’s pretty unlikely. But things like it will happen. People are more likely to give if they’re walking with other people. I infer that they believe they will be abandoned if they do not give.”

Carl: “That’s a far cry from violence.”

Worker: “Think about the context. When you were a baby, you relied on your parents to provide for you, and abandonment by them would have meant certain death. In the environment of evolutionary adaptation, being abandoned by your band would have been close to a death sentence. This isn’t true in the modern world, but people’s brains mostly don’t really distinguish abandonment from violence, and we exploit that.”

Carl: “That makes some sense. I still object to calling it violence, if only because we need a consistent definition of ‘violence’ to coordinate, well, violence against those that are violent. Anyway, I get that this poster is an act, and the things you say to people walking down the street are an act, but what about the charity itself? Do you actually do the things you say you do?”

Worker: “Well, kind of. We actually do give these people cows and stuff, like the poster says. But that isn’t our main focus, and the main reason we do it is, again, because of the act.”

Carl: “Because of the act? Don’t you care about these people?”

Worker: “Kind of. I mean, I do care about them, but I care about myself and my friends more; that’s just how humans work. And if it doesn’t cost me much, I will help them. But I won’t help them if it puts our charity in a significantly worse position.”

Carl: “So you’re the heartless one.”

Worker: “Yes, and so is everyone else. Because the standard you’re set for ‘not heartless’ is not one that any human actually achieves. They just deceive themselves about how much they care about random strangers; the part of their brain that inserts these self-deceptions into their conscious narratives is definitely not especially altruistic!”

Carl: “According to your own poster, there’s going to be famine, though! Is the famine all an act to you?”

Worker: “No! Famine isn’t an act, but most of our activities in relation to it are. We give people cows because that’s one of the standard things charities like ours are supposed to do, and it looks like we’re giving these people local autonomy and stuff.”

Carl: “Looks like? So this is all just optics?”

Worker: “Yes! Exactly!”

Carl: “I’m actually really angry right now. You are a terrible person, and your charity is terrible, and you should die in a fire.”

Worker: “Hey, let’s actually think through this ethical question together. There’s a charity pretty similar to ours that’s set up a stall a couple blocks from here. Have you seen it?”

Carl: “Yes. They do something with water filtering in Africa.”

Worker: “Well, do you think their poster is more or less accurate than ours?”

Carl: “Well, I know yours is a lie, so…”

Worker: “Hold on. This is Gell-Mann amnesia. You know ours is a lie because I told you. This should adjust your model of how charities work in general.”

Carl: “Well, it’s still plausible that they are effective, so I can’t condemn—”

Worker: “Stop. In talking of plausibility rather than probability, you are uncritically participating in the act. You are taking symbols at face value, unless there is clear disproof of them. So you will act like you believe any claim that’s ‘plausible’, in other words one that can’t be disproven from within the act. You have never, at any point, checked whether either charity is doing anything in the actual, material world.”

Carl: “…I suppose so. What’s your point, anyway?”

Worker: “You’re shooting the messenger. All or nearly all of these charities are scams. Believe me, we’ve spent time visiting these other organizations, and they’re universally fraudulent, they just have less self-awareness about it. You’re only morally outraged at the ones that don’t hide it. So your moral outrage optimizes against your own information. By being morally outraged at us, you are asking to be lied to.”

Carl: “Way to blame the victim. You’re the one lying.”

Worker: “We’re part of the same ecosystem. By rewarding a behavior, you cause more of it. By punishing it, you cause less of it. You reward lies that have plausible deniability and punish truth, when that truth is told by sinners. You’re actively encouraging more of the thing that is destroying your own information!”

Carl: “It still seems pretty strange to think that they’re all scams. Like, some of my classmates from college went into the charity sector. And giving cows to people who have food problems actually seems pretty reasonable.”

Worker: “It’s well known by development economists that aid generally creates dependence, that in giving cows to people we disrupt their local economy’s cow market, reducing the incentive to raise cattle. And in theory it could still be worth it, but our preliminary calculations indicate that it probably isn’t.”

Carl: “Hold on. You actually ran the calculation, found that your intervention was net harmful, and then kept doing it?”

Worker: “Yes. Again, it is all—”

Carl: “What the fuck, seriously? You’re a terrible person.”

Worker: “Do you think any charity other than us would have run the calculation we did, and then actually believe the result? Or would they have fudged the numbers here and there, and when even a calculation with fudged numbers indicated that the intervention was ineffective, come up with a reason to discredit this calculation and replace it with a different one that got the result they wanted?”

Carl: “Maybe a few… but I see your point. But there’s a big difference between acting immorally because you deceived yourself, and acting immorally with a clear picture of what you’re doing.”

Worker: “Yes, the second one is much less bad!”

Carl: “What?”

Worker: “All else being equal, it’s better to have clearer beliefs than muddier ones, right?”

Carl: “Yes. But in this case, it’s very clear that the person with the clear picture is acting immorally, while the self-deceiver, uhh..”

Worker: “…has plausible deniability. Their stories are plausible even though they are false, so they have more privilege within the act. They gain privilege by muddying the waters, or in other words, destroying information.”

Carl: “Wait, are you saying self-deception is a choice?”

Worker: “Yes! It’s called ‘motivated cognition’ for a reason. Your brain runs something like a utility-maximization algorithm to tell when and how you should deceive yourself. It’s epistemically correct to take the intentional stance towards this process.”

Carl: “But I don’t have any control over this process!”

Worker: “Not consciously, no. But you can notice the situation you’re in, think about what pressures there are on you to self-deceive, and think about modifying your situation to reduce these pressures. And you can do this to other people, too.”

Carl: “Are you saying everyone is morally obligated to do this?”

Worker: “No, but it might be in your interest, since it increases your capabilities.”

Carl: “Why don’t you just run a more effective charity, and advertise on that? Then you can outcompete the other charities.”

Worker: “That’s not fashionable anymore. The ‘effectiveness’ branding has been tried before; donors are tired of it by now. Perhaps this is partially because there aren’t functional systems that actually check which organizations are effective and which aren’t, so scam charities branding themselves as effective end up outcompeting the actually effective ones. And there are organizations claiming to evaluate charities’ effectiveness, but they’ve largely also become scams by now, for exactly the same reasons. The fashionable branding now is environmentalism.”

Carl: “This is completely disgusting. Fashion doesn’t help people. Your entire sector is morally depraved.”

Worker: “You are entirely correct to be disgusted. This moral depravity is a result of dysfunctional institutions. You can see it outside charity too; schools are authoritarian prisons that don’t even help students learn, courts put people in cages for not spending enough on a lawyer, the US military blows up civilians unnecessarily, and so on. But you already knew all that, and ranting about these things is itself a trope. It is difficult to talk about how broken the systems are without this talking itself being interpreted as merely a cynical act. That’s how deep this goes. Please actually update on this rather than having your eyes glaze over!”

Carl: “How do you even deal with this?”

Worker: “It’s already the reality you’ve lived in your whole life. The only adjustment is to realize it, and be able to talk about it, without this destroying your ability to participate in the act when it’s necessary to do so. Maybe functional information-processing institutions will be built someday, but we are stuck with this situation for now, and we’ll have no hope of building functional institutions if we don’t understand our current situation.”

Carl: “You are wasting so much potential! With your ability to see social reality, you could be doing all kinds of things! If everyone who were as insightful as you were as pathetically lazy as you, there would be no way out of this mess!”

Worker: “Yeah, you’re right about that, and I might do something more ambitious someday, but I don’t really want to right now. So here I am. Anyway… food for the Africans. Helps with local autonomy and environmental sustainability. Have a heart and help them out.”

Carl sighed, fished a 10 dollar bill from his wallet, and gave it to the charity worker.

Decision theory and zero-sum game theory, NP and PSPACE

(Also posted on LessWrong)

At a rough level:

  • Decision theory is about making decisions to maximize some objective function.
  • Zero-sum game theory is about making decisions to optimize some objective function while someone else is making decisions to minimize this objective function.

These are quite different.

Decision theory and NP

Decision theory roughly corresponds to the NP complexity class.  Consider the following problem:

Given a set of items, each of which has a integer-valued value and weight, does there exist a subset with total weight less than w and total value at least v?

(It turns out that finding a solution is not much harder than determining whether there is a solution; if you know how to tell whether there is a solution to arbitrary problems of this form, you can in particular tell if there is a solution that uses any particular item.)

This is the knapsack problem, and it is in NP.  Given a candidate solution, it is easy to check whether it actually is a solution: you just count the values and the weights.  Since this solution would constitute a proof that the answer to the question is “yes”, and a solution exists whenever the answer is “yes”, this problem is in NP.

The following is a general form for NP problems:

\exists x_1 \in \{0, 1\} \exists x_2 \in \{0, 1\} \ldots \exists x_k \in \{0, 1\} f(x_1, ..., x_k)

where f is a specification of a circuit (say, made of AND, OR, and NOT gates) that outputs a single Boolean value.  That is, the problem is to decide whether there is some assignment of values to x_1, \ldots, x_k that f outputs true on.  This is a variant of the Boolean satisfiability problem.

In decision theory (and in NP), all optimization is in the same direction.  The only quantifier is \exists.

Zero-sum game theory and PSPACE

Zero-sum game theory roughly corresponds to the PSPACE complexity class.  Consider the following problem:

Given a specification of a Reversi game state (on an arbitrarily-large square board), does there exists a policy for the light player that guarantees a win?

(It turns out that winning the game is not much harder than determining whether there is a winning policy; if you know how to tell whether there is a solution to arbitrary problems of this form, then in particular you can tell if dark can win given a starting move by light.)

This problem is in PSPACE: it can be solved by a Turing machine using a polynomial amount of space.  This Turing machine works through the minimax algorithm: it simulates all possible games in a backtracking fashion.

The following is a general form for PSPACE problems:

\exists x_1 \in \{0, 1\} \forall y_1 \in \{0, 1\} \ldots \exists x_k \in \{0, 1\} \forall y_k \in \{0, 1\} f(x_1, y_1, \ldots, x_k, y_k)

where f is a specification of a circuit (say, made of AND, OR, and NOT gates) that outputs a single Boolean value.  That is, the problem is to determine whether it is possible to set the x values interleaved with an opponent setting the y values such that, no matter how the opponent acts, f(x_1, y_1, \ldots, x_k, y_k) is true.  This is a variant of the quantified Boolean formula problem.  (Interpreting a logical formula containing \exists and \forall as a game is standard; see game semantics).

In zero-sum game theory, all optimization is in one of two completely opposite directions.  There is literally no difference between something that is good for one player and something that is bad for the other.  The opposing quantifiers \exists and \forall, representing decisions by the two opponents, are interleaved.

Different cognitive modes

The comparison to complexity classes suggests that there are two different cognitive modes for decision theory and zero-sum game theory, as there are two different types of algorithms for NP-like and PSPACE-like problems.

In decision theory, you plan with no regard to any opponents interfering with your plans, allowing you to plan on arbitrarily long time scales.  In zero-sum game theory, you plan on the assumption that your opponent will interfere with your plans (your \existss are interleaved with your opponent’s \foralls), so you can only plan as far as your opponent lacks the ability to interfere with these plans.  You must have a short OODA loop, or your opponent’s interference will make your plans useless.

In decision theory, you can mostly run on naïve expected utility analysis: just do things that seem like they will work.  In zero-sum game theory, you must screen your plans for defensibility: they must be resistant to possible attacks.  Compare farming with border defense, mechanical engineering with computer security.

High-reliability engineering is an intermediate case: designs must be selected to work with high probability across a variety of conditions, but there is normally no intelligent optimization power working against the design.  One could think of nature as an “adversary” selecting some condition to test the design against, and represent this selection by a universal quantifier; however, this is qualitatively different from a true adversary, who applies intentional optimization to break a design rather than haphazard selection of conditions.


These two types of problems do not cover all realistic situations an agent might face.  Decision problems involving agents with different but not completely opposed objective functions are different, as are zero-sum games with more than two players.  But realistic situations share some properties with each of these, and I suspect that there might actually be a discrete distinction between cognitive modes for NP-like decision theory problems and PSPACE-like zero-sum games.

What’s the upshot?  If you want to know what is going on, one of the most important questions (perhaps the most important question) is: what kind of game are you playing?  Is your situation more like a decision theory problem or a zero-sum game?  To what extent is optimization by different agents going in the same direction, opposing directions, or orthogonal directions?  What would have to change for the nature of the game to change?

Thanks to Michael Vassar for drawing my attention to the distinction between decision theory and zero-sum game theory as a distinction between two cognitive modes.

Related: The Face of the Ice

In the presence of disinformation, collective epistemology requires local modeling

In Inadequacy and Modesty, Eliezer describes modest epistemology:

How likely is it that an entire country—one of the world’s most advanced countries—would forego trillions of dollars of real economic growth because their monetary controllers—not politicians, but appointees from the professional elite—were doing something so wrong that even a non-professional could tell? How likely is it that a non-professional could not just suspect that the Bank of Japan was doing something badly wrong, but be confident in that assessment?

Surely it would be more realistic to search for possible reasons why the Bank of Japan might not be as stupid as it seemed, as stupid as some econbloggers were claiming. Possibly Japan’s aging population made growth impossible. Possibly Japan’s massive outstanding government debt made even the slightest inflation too dangerous. Possibly we just aren’t thinking of the complicated reasoning going into the Bank of Japan’s decision.

Surely some humility is appropriate when criticizing the elite decision-makers governing the Bank of Japan. What if it’s you, and not the professional economists making these decisions, who have failed to grasp the relevant economic considerations?

I’ll refer to this genre of arguments as “modest epistemology.”

I see modest epistemology as attempting to defer to a canonical perspective: a way of making judgments that is a Schelling point for coordination. In this case, the Bank of Japan has more claim to canonicity than Eliezer does regarding claims about Japan’s economy. I think deferring to a canonical perspective is key to how modest epistemology functions and why people find it appealing.

In social groups such as effective altruism, canonicity is useful when it allows for better coordination. If everyone can agree that charity X is the best charity, then it is possible to punish those who do not donate to charity X. This is similar to law: if a legal court makes a judgment that is not overturned, that judgment must be obeyed by anyone who does not want to be punished. Similarly, in discourse, it is often useful to punish crackpots by requiring deference to a canonical scientific judgment.

It is natural that deferring to a canonical perspective would be psychologically appealing, since it offers a low likelihood of being punished for deviating while allowing deviants to be punished, creating a sense of unity and certainty.

An obstacle to canonical perspectives is that epistemology requires using local information. Suppose I saw Bob steal my wallet. I have information about whether he actually stole my wallet (namely, my observation of the theft) that no one else has. If I tell others that Bob stole my wallet, they might or might not believe me depending on how much they trust me, as there is some chance I am lying to them. Constructing a more canonical perspective (e.g. a in a court of law) requires integrating this local information: for example, I might tell the judge that Bob stole my wallet, and my friends might vouch for my character.

If humanity formed a collective superintelligence that integrated local information into a canonical perspective at the speed of light using sensible rules (e.g. something similar to Bayesianism), then there would be little need to exploit local information except to transmit it to this collective superintelligence. Obviously, this hasn’t happened yet. Collective superintelligences made of humans must transmit information at the speed of human communication rather than the speed of light.

In addition to limits on communication speed, collective superintelligences made of humans have another difficulty: they must prevent and detect disinformation. People on the internet sometimes lie, as do people off the internet. Self-deception is effectively another form of deception, and is extremely common as explained in The Elephant in the Brain.

Mostly because of this, current collective superintelligences leave much to be desired. As Jordan Greenhall writes in this post:

Take a look at Syria. What exactly is happening? With just a little bit of looking, I’ve found at least six radically different and plausible narratives:

• Assad used poison gas on his people and the United States bombed his airbase in a measured response.

• Assad attacked a rebel base that was unexpectedly storing poison gas and Trump bombed his airbase for political reasons.

• The Deep State in the United States is responsible for a “false flag” use of poison gas in order to undermine the Trump Insurgency.

• The Russians are responsible for a “false flag” use of poison gas in order to undermine the Deep State.

• Putin and Trump collaborated on a “false flag” in order to distract from “Russiagate.”

• Someone else (China? Israel? Iran?) is responsible for a “false flag” for purposes unknown.

And, just to make sure we really grasp the level of non-sense:

• There was no poison gas attack, the “white helmets” are fake news for purposes unknown and everyone who is in a position to know is spinning their own version of events for their own purposes.

Think this last one is implausible? Are you sure? Are you sure you know the current limits of the war on sensemaking? Of sock puppets and cognitive hacking and weaponized memetics?

All I am certain of about Syria is that I really have no fucking idea what is going on. And that this state of affairs — this increasingly generalized condition of complete disorientation — is untenable.

We are in a collective condition of fog of war. Acting effectively under fog of war requires exploiting local information before it has been integrated into a canonical perspective. In military contexts, units must make decisions before contacting a central base using information and models only available to them. Syrians must decide whether to flee based on their own observations, observations of those they trust, and trustworthy local media. Americans making voting decisions based on Syria must decide which media sources they trust most, or actually visit Syria to gain additional info.

While I have mostly discussed differences in information between people, there are also differences in reasoning ability and willingness to use reason. Most people most of the time aren’t even modeling things for themselves, but are instead parroting socially acceptable opinions. The products of reasoning could perhaps be considered as a form of logical information and treated similar to other information.

In the past, I have found modest epistemology aesthetically appealing on the basis that sufficient coordination would lead to a single canonical perspective that you can increase your average accuracy by deferring to (as explained in this post). Since then, aesthetic intuitions have led me to instead think of the problem of collective epistemology as one of decentralized coordination: how can good-faith actors reason and act well as a collective superintelligence in conditions of fog of war, where deception is prevalent and creation of common knowledge is difficult? I find this framing of collective epistemology more beautiful than the idea of a immediately deferring to a canonical perspective, and it is a better fit for the real world.

I haven’t completely thought through the implications of this framing (that would be impossible), but so far my thinking has suggested a number of heuristics for group epistemology:

  • Think for yourself. When your information sources are not already doing a good job of informing you, gathering your own information and forming your own models can improve your accuracy and tell you which information sources are most trustworthy. Outperforming experts often doesn’t require complex models or extraordinary insight; see this review of Superforecasting for a description of some of what good amateur forecasters do.
  • Share the products of your thinking. Where possible, share not only opinions but also the information or model that caused you to form the opinion. This allows others to verify and build on your information and models rather than just memorizing “X person believes Y”, resulting in more information transfer. For example, fact posts will generally be better for collective epistemology than a similar post with fewer facts; they will let readers form their own models based on the info and have higher confidence in these models.
  • Fact-check information people share by cross-checking it against other sources of information and models. The more this shared information is fact-checked, the more reliably true it will be. (When someone is wrong on the internet, this is actually a problem worth fixing).
  • Try to make information and models common knowledge among a group when possible, so they can be integrated into a canonical perspective. This allows the group to build on this, rather than having to re-derive or re-state it repeatedly. Contributing to a written canon that some group of people is expected to have read is a great way to do this.
  • When contributing to a canon, seek strong and clear evidence where possible. This can result in a question being definitively settled, which is great for the group’s ability to reliably get the right answer to the question, rather than having a range of “acceptable” answers that will be chosen from based on factors other than accuracy.
  • When taking actions (e.g. making bets), use local information available only to you or a small number of others, not only canonical information. For example, when picking organizations to support, use information you have about these organizations (e.g. information about the competence of people working at this charity) even if not everyone else has this info. (For a more obvious example to illustrate the principle: if I saw Bob steal my wallet, then it’s in my interest to guard my possessions more closely around Bob than I otherwise would, even if I can’t convince everyone that Bob stole my wallet).

Against unreasonably high standards

Consider the following procedure:

  1. Create unreasonably high standards that people are supposed to follow.
  2. Watch as people fail to meet them and thereby accumulate “debt”.
  3. Provide a way for people to discharge their debt by sacrificing their agency to some entity (concrete or abstract).

This is a common way to subjugate people and extract resources from them.  Some examples:

  • Christianity: Christianity defines many natural human emotions and actions as “sins” (i.e. things that accumulate debt), such that almost all Christians sin frequently.  Even those who follow all the rules have “original sin”.  Christianity allows people to discharge their debt by asking Jesus to bear their sins (thus becoming subservient to Jesus/God).
  • The Western education system: Western schools (and many non-Western schools) create unnatural standards of behavior that are hard for students to follow.  When students fail to meet these standards, they are told they deserve punishments including public humiliation and being poor as an adult.  School doesn’t give a way to fully discharge debts, leading to anxiety and depression in many students and former students, but people can partially discharge debt by admitting that they are in an important sense subservient to the education system (e.g. accepting domination from the more-educated boss in the workplace).
  • Effective altruism: The drowning child argument (promoted by effective altruists such as Peter Singer) argues that middle-class Americans have an obligation to sacrifice luxuries to save the lives of children in developing countries, or do something at least this effective (in practice, many effective altruists instead support animal welfare or existential risk organizations).  This is an unreasonably high standard; nearly no one actually sacrifices all their luxuries (living in poverty) to give away more money.  Effective altruism gives a way to discharge this debt: you can just donate 10% of your income to an effective charity (sacrificing some of your agency to it), or change your career to a more good-doing one.  (This doesn’t work for everyone, and many “hardcore EAs” continue to struggle with scrupulosity despite donating much more than 10% of their income or changing their career plans significantly, since they always could be doing more).
  • The rationalist community: I hesitate to write this section for a few reasons (specifically, it’s pretty close to home and is somewhat less clear given that some rationalists have usefully criticized some of the dynamics I’m complaining about).  But a subtext I see in the rationalist community says something like: “You’re biased so you’re likely to be wrong and make bad decisions that harm other people if you take actions in the world, and it’ll be your fault.  Also, the world is on fire and you’re one of the few people who knows about this, so it’s your responsibility to do something about it.  Luckily, you can discharge some of your debts by improving your own rationality, following the advice of high-level rationalists, and perhaps giving them money.”  That’s clearly an instance of this pattern; no one is unbiased, “high-level rationalists” included.  (It’s hard to say where exactly this subtext comes from, and I don’t think it’s anyone’s “fault”, but it definitely seems to exist; I’ve been affected by it myself, and I think it’s part of what causes akrasia in many rationalists.)

There are many more examples; I’m sure you can think of some.  Setting up a system like this has some effects:

  • Hypocrisy: Almost no one actually follows the standards, but they sometimes pretend they do.  Since standards are unreasonably high, they are enforced inconsistently, often against the most-vulnerable members of a group, while the less-vulnerable maintain the illusion that they are actually following the standards.
  • Self-violence: Buying into unreasonably high standards will make someone turn their mind against itself.  Their mind will split between the “righteous” part that is trying to follow and enforce the unreasonably high standards, and the “sinful” part that is covertly disobeying these standards in order to get what the mind actually wants (which is often in conflict with the standards).  Through neglect and self-violence, the “sinful” part of the mind develops into a shadow.  Self-hatred is a natural results of this process.
  • Distorted perception and cognition: The righteous part of the mind sometimes has trouble looking at ways in which the person is failing to meet standards (e.g. it will avoid looking at things that the person might be responsible for fixing).  Consciousness will dim when there’s risk of seeing that one is not meeting the standards (and sometimes also when there’s risk of seeing that others are not meeting the standards).  Concretely, one can imagine someone who gets lost surfing the internet to avoid facing some difficult work they’re supposed to do, or someone who avoids thinking about the ways in which their project is likely to fail.  Given the extent of the high standards and the debt that most people feel they are in, this will often lead to extremely distorted perception and cognition, such that coming out of it feels like waking from a dream.
  • Motivational problems: Working is one way to discharge debt, but working is less motivating if all products of your work go to debt-collectors rather than yourself.  The “sinful” part of the mind will resist work, as it expects to derive little benefit from it.
  • Fear: Accumulating lots of debt gives one the feeling that, at any time, debt-collectors could come and demand anything of you.  This causes the scrupulous to live in fear.  Sometimes, there isn’t even a concretely-identifiable entity they’re afraid of, but it’s clear that they’re afraid of something.

Systems involving unreasonably high standards could theoretically be justified if they were good coordination mechanisms.  But it seems implausible that they are.  Why not just make the de jure norms ones that people are actually likely to follow?  Surely a sufficient set of norms exists, since people are already following the de facto ones.  You can coordinate a lot without optimizing your coordination mechanism for putting everyone in debt!

I take the radical position that TAKING UNREASONABLY HIGH STANDARDS SERIOUSLY IS A REALLY BAD IDEA and ALL OF MY FRIENDS AND PERHAPS ALL HUMANS SHOULD STOP DOING IT.  Unreasonably high standards are responsible for a great deal of violence against life, epistemic problems, and horribleness in general.

(It’s important to distinguish having unreasonably high standards from having a preference ordering whose most-preferred state is impractical to attain; the second does not lead to the same problems unless there’s some way of obligating people to reach an unreasonably good state in the preference ordering.  Attaining a decent but non-maximally-preferred state should perhaps feel annoying or aesthetically displeasing, but not anxiety-inducing.)

My advice to the scrupulous: you are being scammed and you are giving your life away to scammers.  The debts that are part of this scam are fake, and you can safely ignore almost all of them since they won’t actually be enforced.  The best way to make the world better involves first refusing to be scammed, so that you can benefit from the products of your own labor (thereby developing intrinsic motivation to do useful things) instead of using them to pay imaginary debts, and so you can perceive the world accurately without fear.  You almost certainly have significant intrinsic motivation for helping others; you are more likely to successfully help them if your help comes from intrinsic motivation and abundance rather than fear and obligation.

Coordination isn’t hard

In discussions around existential risk, I sometimes hear the phrase “coordination is hard”.  Most of the time, the person saying this is using this as a reason to give up on strategies involving decentralized coordination and instead pursue more-centralized strategies, such as “a small number of people being extremely rational individuals and using other people as advisers and actuators”.  In this context, I think the phrase “coordination is hard” is very misleading.

Here are some examples of successful coordination:

  • Labor movements manage to hold strikes despite individual incentives to break the strike.
  • Open source software is usually created by volunteers and is often of fairly high quantity and quality.  Some projects, such as Linux, have hundreds or thousands of contributors.
  • CFCs have been largely phased out globally due to their impact on the ozone layer.
  • Communities, in-person and online, develop discourse norms, which can implement significant functionality for the community (e.g. fact-checking, preventing people from unjustly harming others)
  • Some groups of people distributed among different countries (e.g. workers, moderates, various racial/ethnic groups, intellectuals) will feel affinity with each other and help each other out (e.g. intellectuals in the US might support immigration as a way of assisting intellectuals in other countries)
  • Militaries usually have significant decentralized coordination.  Soldiers usually fight bravely; defectors are relatively rare.
  • Many societies have a well-functioning justice system.  This system requires people to perform many different functions (judge, juror, witness, expert, …).

None of these examples are in perfect analogy with current existential risk issues, but given them, the generic statement “coordination is hard” looks silly.  Upon dismissing this generic statement, more interesting questions emerge:

  • When is coordination easy or hard?
  • How does ability to coordinate vary across different societies historically?
  • What different coordination strategies are there?  How do they work?  What are their strengths and weaknesses?

In many cases, humans seem to be better at coordinating than Homo economicus would predict; anyone concerned with the future of humanity ought to find this fact extremely interesting.





Naïve epistemology, savvy epistemology

[epistemic status: confident this distinction exists and is important, unsure about details/framing]

A contrast:

  • Naïve epistemology: simple models; mathematics and logic; “prima facie”; original seeing; “natural” descriptions of facts on the ground; “from the mouth of a babe”; not trusting others’ models/narratives (but often trusting their reports of basic observable facts); blunt
  • Savvy epistemology: responding to the dominant opinions in a social group such as expert consensus (agreeing with parts of it and/or disagreeing with parts of it); “this is a complex topic”; not wanting to omit considerations that others know about; making sure to be “informed” and “up to date”; educated; responsible; legitimate; sophisticated; subtle; trust or cynicism regarding others’ models/narratives

Naïve epistemology mostly models the world; savvy epistemology mostly models other people’s models of the world.  Naïve epistemology avoids the social field; savvy epistemology swims in it.

The virtues of naiveté

In the past year, I’ve strongly updated towards using more naïve epistemology than savvy epistemology.  Savvy epistemology is more informed in some ways, but is also more corrupt.  It will tend to be slanted by whoever has local political power, rather than being purely optimized for accuracy.  Some examples:

  • Tetlock’s book Superforecasting shows that some non-experts using relatively naïve methods can predict world events significantly better than savvier experts.  This indicates that these experts’ opinions are not very well optimized for accuracy (and are most likely optimized for something other than accuracy).
  • I am convinced by Robin Hanson’s argument that medical care has little association with health outcomes in the US.  The “savvy” responses to his “naïve” article (linked at the bottom of the page) seem to basically be misdirections intended to avoid taking seriously his main point (that cutting medical spending by 50% across the board would make little difference to health outcomes).
  • AI risk was discussed on LessWrong many years before the topic received serious mainstream attention, mostly from a naïve epistemology perspective.  The savvy expert view started out saying it wasn’t a serious problem (for various reasons), and has since drifted somewhat towards the LessWrong view.  From thinking extensively about AI alignment and talking with people over the past few years, I feel that I have gotten significant personal evidence in favor of the LessWrong view that the alignment problem is very difficult and that no one has a good idea of how to solve it.

The corruption of savvy epistemology does not make it useless: it usually contains a great deal of information not captured by naïve epistemology, and use of savvy epistemology allows for easier coordination with others using it.  But it does mean that it’s highly valuable to develop a naïve perspective in addition to a savvy one.

Why be afraid of naïve epistemology?

There are legitimate reasons to be afraid that someone taking naïve epistemology seriously will cause harm:

  • Uncooperative actions: Taking one’s own models seriously can lead one to take actions that are uncooperative with others who have different models, such as dishonesty or violence.  See The Unilateralist’s Curse.  The “social field” that savviness swims in is a coordination mechanism that, among other things, prevents people from taking some uncooperative actions.
  • Safety: Common recommendations for what to believe, though often false, at least offer safety in the medium term for most people.  Having and acting on unusual beliefs can be dangerous to one’s self and others, since at that point you’re not using a set of beliefs that’s already been optimized for medium-term safety.  (In the long term, safety from some risks such as large-scale wars and some medical issues might require nonconformist epistemics)
  • Disharmony: To the extent that words ostensibly about the world are used to coordinate actions, people using different words (e.g. those generated by naïve epistemology) can generate confusion about what actions the group is supposed to be taking.
  • Politics: Saying naïve things can disrupt politics, by e.g. offending allies or potential allies.  This can trigger a defensive reaction.
  • Legibility bias: Naïve models tend to be legible.  It can be tempting to make oversimplified models that don’t actually summarize the fiddly details well, leading to failures such as utopian totalitarianism.

Social technology for being naïve

I think that people who want good epistemics should have the affordance to use and act on naïve epistemology.  This requires addressing the problems in the previous section.  Here are some tools I use to give myself the affordance to do naïve epistemology without paying large costs:

  • Decision theory: Preferentially use naïve epistemology when making bets using one’s own resources, and use shared epistemology when making bets using others’ resources.  Detect “prisoner’s dilemma” or “unilateralist’s curse” situations (including epistemic ones) and make a good-faith effort to achieve mutual cooperation.  Develop and signal the intent to honestly inform others and to learn from others.  Take the true outside view and act as you would want others in your position to act.  (See also: Guided by the beauty of our weapons)
  • Testing things: If a naïve model implies some surprising prediction, it can be good to test it out (if it isn’t very costly).  If the prediction is confirmed, great; if not, then perhaps create a new naïve model that takes this new piece of evidence into account.
  • Separate magisteria: Calling forms of model-building “naïve” puts these models in a separate magisterium where it isn’t socially expected that they will be taken seriously, triggering less defensive reaction.  (I bet many people would be less receptive to this post if I had called it “world-modeling epistemology” rather than “naïve epistemology”).  Other ways of putting model-building in a separate magisterium: calling yourself “a contrarian acting on a weird inside view”; saying you’re “discussing weird ideas”; etc.
  • Translation: Some naïve models can eventually be translated into savvy arguments.  (In practice, not all of them can be translated without losing fidelity).
  • Arguing against savviness:  In general, internalizing arguments against “mainstream” opinions makes it psychologically and socially easier to do “non-mainstream” things.  This post itself contains arguments against savvy epistemology.  Getting arguments against specific savvy beliefs/worldviews can also be helpful.
  • Playing with identity: In general, some identities (“independent”, “truth-seeking”, “weird”) will create a local incentive to lean towards naïve epistemology,  while others (“responsible”, “official”, “educated”) will create a local incentive to lean towards savvy epistemology.  Someone who leans savvy might benefit by creating a context where they can embody a more naïve identity.

I would like it if more people had the affordance to develop a naïve epistemology, so that they can make important inferences that savvy epistemology obscures.  I don’t think that naïve epistemology is an “endpoint” of epistemology (it seems desirable to eventually develop epistemology that is fundamentally truth-tracking the way naïve epistemology is in addition to being highly informed and coordinated across people), but I expect it to be an important stepping stone for people currently embedded in savvy epistemology.

Beware defensibility

[epistemic status: confident this is a thing, not sure if I’m framing it right]

I’ve noticed that my writing is usually better when I write it quickly and don’t edit it as much.  Continuously adjusting it results in something blander.  I think this is about defensibility.  Some examples:

  • Academic papers are more defensible than blog posts.
  • Public relations speech is more defensible that private speech.
  • Proofs are more defensible than intuitive arguments.
  • Slowly-written things are more defensible than quickly-written things.
  • “Literal” speech is more defensible than “metaphorical” speech.

Defensibility is about resisting all possible attacks from some class of adversaries.  The more you expect your expression to be picked apart and used against you, the more defensible your expression will be.  Slowly-written things look more polished and the flaws stand out more, so they get nitpicked.

Defensibility can be good.  In mathematics, a defensible argument (i.e. a proof) is more likely to be correct.  In science, a defensible statistical result is more likely to replicate.  Defensible results become blocks of knowledge that others can (and sometimes must) build upon.

Defensibility can be bad.  Academic papers are usually worse at explaining things than blog posts.  PR is usually highly misleading.  Fully formal proofs are usually harder to understand than intuitive arguments.  Defensible art is bland.

Defensibility requires conformity.  If anything you say can and will be used against you, it is better to say the same things others are saying.  Defensible expressions happen in a shared ontology, such as formal logic, or the “ordinary official speech” ontology that Wikipedia uses.

Some expression should be defensible (against different classes of adversaries).  Some shouldn’t be.  It is virtuous to be flexible about defensibility.

Rationality techniques as patterns

[epistemic status: optimistic vision that currently seems highly useful]

In Christopher Alexander’s work, a pattern consists of:

  1. A way of perceiving an existing tension in a living system.
  2. A model for why the tension exists.
  3. A way of adapting the system in a way that resolves the tension.

Christopher Alexander describes hundreds of patterns for towns and buildings in A Pattern Language.  Consider pattern #126, SOMETHING ROUGHLY IN THE MIDDLE:

A public space without a middle is quite likely to stay empty.

if there is a reasonable area in the middle, intended for public use, it will be wasted unless there are trees, monuments, seats, fountains—a place where people can protect their backs, as easily as they can around the edge.


Between the natural paths which cross a public square or courtyard or piece of common land choose something to stand roughly in the middle: a fountain, a tree, a statue, a clock-tower with seats, a windmill, a bandstand.  Make it something which gives a strong and steady pulse to the square, drawing people in twoards the center.  Leave it exactly where it falls between the paths; resist the impulse to put it exactly in the middle.

This pattern points out a tension: a public space without a middle will feel weird and people will avoid it.  Next, it gives a model for the tension: it’s because people want their back to be against something.  Finally, it gives a way to resolve this tension: put something roughly in the middle.

Patterns that work well with each other form a pattern language.  SOMETHING ROUGHLY IN THE MIDDLE works with SMALL PUBLIC SQUARES; along with the other patterns, they form a pattern language.

The initial tension will sometimes be diffuse and hard to pin down.  People might feel “off” when they’re in a large empty space but not have the words to say why.  They might not even consciously realize there is a problem and instead just have slightly elevated stress levels.  The tension becomes more pointed if it is explicitly pointed out and especially if there is a model for it.

Many of the rationality techniques discussed in the LessWrong-descended rationality community are patterns in this sense.  The technique of leaving a line of retreat is about noticing a tension (some possibility X seems hard to seriously consider) and resolving it by imagining a world in which X is true in detail (in order to allow yourself to compare the world where X is true and the world where X is false even-handedly).  The technique of finding a true rejection is about noticing a tension (people have arguments where they give false reasons for their beliefs and don’t actually change their minds when these reasons are refuted) and resolving it by urging people to find the actual historical reasons for their beliefs.  CFAR‘s debugging techniques are also usually patterns of this form (they resolve some sort of tension within a person or between a person and their environment, usually reducing stress levels).

Resolving tensions fully brings to mind the image of a placid lake.  But resolving tensions doesn’t necessarily lead to placidity, because systems involving humans are alive.  Solving all of someone’s immediate problems might make them bored, and might make them want to help others.

Christopher Alexander praises the way that well-built towns and buildings bring out life in The Timeless Way of Building (I highly recommend reading the first 2 chapters):

There is one timeless way of building.

It is thousands of years old, and the same today as it has always been.

The great traditional buildings of the past, the villages and tents and temples in which man feels at home, have always been made by people who were very close to the center of this way.  It is not possible to make great buildings, or great towns, beautiful places, places where you feel yourself, places where you feel alive, except by following this way.  And, as you will see, this way will lead anyone who looks for it to buildings which are themselves as ancient in their form, as the trees and hills, and as our faces are.

It is a process through which the order of a building or a town grows out directly from the inner nature of the people, and the animals, and plants, and matter which are in it.

It is a process which allows the life inside a person, or a family, or a town, to flourish, openly, in freedom, so vividly that it gives birth, of its own accord, to the natural order which is needed to sustain this life.

But as things are, we have so far beset ourselves with rules, and concepts, and ideas of what must be done to make a building or a town alive, that we have become afraid of what will happen naturally, and convinced that we must work within a “system” and with “methods” since without them our surroundings will come tumbling down in chaos.

The thoughts and fears which feed these methods are illusions.

I find Christopher Alexander’s vision highly appealing.  Something happened in Western architecture that interfered with the timeless way of building; this probably had to do with the increase in degrees of freedom in the transition from pre-modern to modern societies.  Similar phenomena have happened in other domains.  Luckily, the way can be regained through the discipline of pattern languages, and eventually through dissolving disciplines and doing what is natural.  There are obvious parallels with Taoism.

I want pattern languages for improving people’s psychological functioning, for improving groups’ ability to work together, for improving models of the world, for making good institutions, for all sorts of things.  Groups of people are developing pattern languages in many different places, mostly without explicitly knowing what they are doing.  I think a more explicit understanding of pattern languages and knowledge of how past pattern languages were created will help to design better ones in the future.  Maybe with the right pattern languages (and with the right going-beyond-them), we can become much more alive, as individuals and as groups, and much more able to coordinate to accomplish very hard things.

The true outside view

[epistemic status: strong opinions weakly held, not very original]

The false outside view says that if you have a contrarian inside view, you’re probably wrong, so you should not act like you’re correct. The false outside view is driven by intuitions about social status and/or fear of being wrong.

The true outside view says that a great deal of good comes from contrarians pushing on their contrarian inside views (acting on them, talking about them with others, etc), so if you have one of those you should maybe do that, while additionally taking others’ perspectives seriously, testing your ideas often, being honest, avoiding some potentially-catastrophic unilateral actions, etc.

The true outside view follows from any halfway-decent decision theory (something like “updateless decision theory for humans”, as gestured at before. This decision theory might look “epistemically modest” sometimes and “epistemically arrogant” other times, but these descriptions impose ego on something that by construction doesn’t have an ego.