Imagine you have magic mirror, capable of answering yes/no questions from 3 years time with perfect calibration. What would you do with it?
I was having drinks with a friend the other day and she asked why I respected Xi, the Chinese Premier. I started by saying that the Chinese bureaucracy seems pretty sharp and it won’t get taken over by just anyone. Then I said that honestly I wasn’t sure because I think I really don’t know that much. And finally I looked up some prediction markets.
There are a lot of prediction markets on Manifold about China, but they didn’t provide a picture. Manifold is pretty well-calibrated too so that wasn’t the issue. When I think about China in 5 years it’s like I’m trying to describe the divine. My mind knows some things but they seem like the musings of a child. China will probably still exist. Xi will probably still be in charge. China is big and complex. Hey, at least my mind doesn’t bullshit me.
So we have something like a tool for predicting the future, but it isn’t useful for me in basic discussions. What is going on?
Let’s imagine had a much better tool - we can predict the next few years well and cheaply. What would we do?
Play the stock market. For our purposes, this is a non-answer. It’s already happening. Hedge funds have all the incentives and talent to do this and their information gives us accurate prices.
Add it to wikipedia. Currently, wikipedia only allows for news articles and so can only discuss the present and the past. But if we had an oracle, it could discuss the future too “America is the largest economy in the world [citation from the world bank] and seems likely to stay so until 2034 when on balance, china may take over [citation from the oracle]”.
This assumes that the issue we currently have is that we aren’t taking pictures of the future but individual pixels. Framing it as a wikipedia article might solve this, forcing us to ask questions to fill in the blanks. It would be obvious how little we know about each region of China, each key decision maker, each export commodity. The reason I speak like a child is because my probability tree is basically a single node with China go good? written on it. Until that changes, I might not be able to have a better discussion about China, oracle or no.1
We could work closely with decision makers. Currently many questions are so narrow as to be useless. When I worked in the UK Civil Service, I would not have wanted “Will the UK grow by more than 1% this year” I would have wanted “If you allocate space around Allsmouth port according to an auction, how many high paying local jobs will it create before 2027? Will there be bad publicity?” But that would have required me to write the questions to the oracle and then maybe ask another set once it was answered. The Swift Centre, who I forecast for, is doing work like this, running live forecasts of breaking news2.
It seems plausible that if we had a magical oracle we could sit with decision makers for a day pinging their questions back and forth. In particular, we’d need to know what they planned to do, then forecast around that and see if it changed their plan. Only once we really knew what they intended could we start to scope further into the future. This process invisages the future as a set of directions. We have to figure out what someone is going to do first before we can even give a direction. What direction is the car currently moving? Will you take the next left, or not? Instead forecasting often just gives a running commentary of the distance from the moon, as if that would help you drive at all.
We could use it as a mediation tool. For instance, the U.S. and China could both agree that neither of them wants large-scale or nuclear conflict. The oracle could then predict how certain actions, like arms buildup or lack of a nuclear treaty, would make such conflicts more likely. Perhaps there was no way to avoid arms buildup, but it seem like you could predict the success of different types of anti-nuclear treaty. You could ask the parties what they might sign and push that back into the oracle to get better options.
This is pretty similar to the discussion with decision makers but is more coarsely grained. It asks “What are outcomes we all really want or all really hate? Are there any options we all find acceptable which get us there?” This becomes less a discussion of the general picture and more “if there is one thing we can do, it’s…” Even this would have been useful in discussing with my friend - I don’t know much about China but I know we need to avoid a nuclear war.
I think it’s important to notice when things aren’t working. I spend a lot of time on forecasting aggregators (eg Metaculus) and prediction markets, but I think it’s not really helping me make better decisions. How could that change?
Can you think of any others? I have a work in progress version of this on my website, here so if you see this a long time after publishing, it may have changed.3
If you are interested in funding a project like this, email me. I guess it will take about $100k to see some initial progress, either working with the LessWrong wiki or Manifold
I am biased but I think the Swift Centre is building some of the most interesting processes in forecasting right now - tight loops between forecasters and decision makers. Over time I have become less optimistic about large public forecasts/prediction markets and more optimistic about this private work. One route to useful forecasts is a lot of input from those who will use them.
I think a subtitle of this article “god knows, omniscience isn’t enough” is funnier, but more confusing. God does, if He exists, know this.
Forecast sharpness is massively undervalued here. Insufficiently sharp forecasts are extremely hard to make decisions from. In fact, in many cases, trading some calibration for sharpness might be preferable.
As an extreme example, assume you can choose between two oracles for binary outcomes. One is perfectly calibrated but always gives probabilities between 40 and 60%. The other only ever predicts 1 or 99% but its calibration is slightly off. Assuming "slightly" isn't too big, the latter oracle is probably more useful for many real decision.