Base rates: Question category has a base rate.
* Conservative (30-70% band) War, Treaties & International Agreements, Economics, International Conflict, Elections
* Moderate: (20-80% band) Markets, Interest Rates, Instability, Indicators, Health, Exchange Rates, Finance
* Aggressive: (5%-95% band) Leaders, Weapons, Taxes, Legislation, International Organizations, Foreign Policy, Environment, Energy, Domestic Politics, Commodities
Stick with the category & question base rates, or risk making stupid mistakes.
When going contrary corner, go conservative.
Questions
* Can the question be parsed into parts that can be assessed individually for risk?
* Is the question framed with very specific resolution criteria? If so, weigh against.
* Review the scoring criteria. If ordered, spread the probabilities over more likely options.
* Attempt to assess how much is unknown or a function of random chance. If there is no base rate and uncertainty, then stay between 30-70%.
* Look for recent trends and incorporate them into your forecast.
Process
* Always review initial base rate before updating and factor that into your forecast.
* Automate forecasts and/or reassessments wherever possible.
* Is there a frequently updated information source that can serve as a proxy for the metric being forecasted? For example, aidworkersecurity.org/incidents is updated frequently and parallels the information in Aid in Danger or the ten year bond rate for a country for the rate in OECD.
* Even with database update variability, base rate is better than downward weights.
* Autocratic governments have less variability. Go with the status quo. Weak democracies mean more variability. Hedge.
* Do not make late calls on back-dated questions. It’s all risk and upside is limited.
* Find ways to eliminate options in questions with more options.
* Check for overconfidence.
Misc
* The base rate for bills passing U.S. Congress in the last decade is 1-3%.
* The base rate for U.S. state legislatures range from <10% to as much as 50% in the Western States,
which pass few bills comparatively.
* U.S. State legislatures are probably more representative of international, which should probably have a cushion built in because there are many unknowns, say 15-30% chance of a non-U.S. bill passing into law.
* The politics of countries you do not live in are even more difficult to forecast. Hedge bets.
* People’s expressed desires in media may not reflect what they really think.
* When deriving a base rate from a small sample size, such as the length of Special Counsel investigations, it is best to build in a big cushion.
* Regulation slows the deployment of fundamentally new technology and like laws, new regulations are slow to develop.
* Be careful when going low or against the crowd on a popular franchise.
* There is a difference between technically possible and a product launch of that capability.
* It is easy to think that there is something special in the moment, that make this situation different and it is going to remake the landscape, e.g., the trend toward majority governments in first-past-the-post-systems or a new party will completely change the political landscape. It happens, but rarely. Forecasts should bet against the rare occurrence unless there is a very good reason to do so.
* Questions that resolve around politics, popular opinion, or the opinions of individuals or a select group of individuals need a significant hedge because there are always significant factors that are in play that cannot be accounted for in a forecast.
* Know the technical details, such as laws and rules. They are often crucial.
* Know when you aren’t bringing anything useful to the prediction and the result is random chance.
* Make sure you have some probabilistic edge on every question, even if it is going with default when the crowd is making a risky call.
* It is very difficult to come up with a predictive model for subjective preferences.
* If someone is in the media and subsequently gets into politics, give a boost to their probabilities due to para-social relationships.
* Don’t exclusively follow others. Always do your own research.
* Luck counts too.
Base rates: Question category has a base rate.
* Conservative (30-70% band) War, Treaties & International Agreements, Economics, International Conflict, Elections
* Moderate: (20-80% band) Markets, Interest Rates, Instability, Indicators, Health, Exchange Rates, Finance
* Aggressive: (5%-95% band) Leaders, Weapons, Taxes, Legislation, International Organizations, Foreign Policy, Environment, Energy, Domestic Politics, Commodities
Stick with the category & question base rates, or risk making stupid mistakes.
When going contrary corner, go conservative.
Questions
* Can the question be parsed into parts that can be assessed individually for risk?
* Is the question framed with very specific resolution criteria? If so, weigh against.
* Review the scoring criteria. If ordered, spread the probabilities over more likely options.
* Attempt to assess how much is unknown or a function of random chance. If there is no base rate and uncertainty, then stay between 30-70%.
* Look for recent trends and incorporate them into your forecast.
Process
* Always review initial base rate before updating and factor that into your forecast.
* Automate forecasts and/or reassessments wherever possible.
* Is there a frequently updated information source that can serve as a proxy for the metric being forecasted? For example, aidworkersecurity.org/incidents is updated frequently and parallels the information in Aid in Danger or the ten year bond rate for a country for the rate in OECD.
* Even with database update variability, base rate is better than downward weights.
* Autocratic governments have less variability. Go with the status quo. Weak democracies mean more variability. Hedge.
* Do not make late calls on back-dated questions. It’s all risk and upside is limited.
* Find ways to eliminate options in questions with more options.
* Check for overconfidence.
Misc
* The base rate for bills passing U.S. Congress in the last decade is 1-3%.
* The base rate for U.S. state legislatures range from <10% to as much as 50% in the Western States,
which pass few bills comparatively.
* U.S. State legislatures are probably more representative of international, which should probably have a cushion built in because there are many unknowns, say 15-30% chance of a non-U.S. bill passing into law.
* The politics of countries you do not live in are even more difficult to forecast. Hedge bets.
* People’s expressed desires in media may not reflect what they really think.
* When deriving a base rate from a small sample size, such as the length of Special Counsel investigations, it is best to build in a big cushion.
* Regulation slows the deployment of fundamentally new technology and like laws, new regulations are slow to develop.
* Be careful when going low or against the crowd on a popular franchise.
* There is a difference between technically possible and a product launch of that capability.
* It is easy to think that there is something special in the moment, that make this situation different and it is going to remake the landscape, e.g., the trend toward majority governments in first-past-the-post-systems or a new party will completely change the political landscape. It happens, but rarely. Forecasts should bet against the rare occurrence unless there is a very good reason to do so.
* Questions that resolve around politics, popular opinion, or the opinions of individuals or a select group of individuals need a significant hedge because there are always significant factors that are in play that cannot be accounted for in a forecast.
* Know the technical details, such as laws and rules. They are often crucial.
* Know when you aren’t bringing anything useful to the prediction and the result is random chance.
* Make sure you have some probabilistic edge on every question, even if it is going with default when the crowd is making a risky call.
* It is very difficult to come up with a predictive model for subjective preferences.
* If someone is in the media and subsequently gets into politics, give a boost to their probabilities due to para-social relationships.
* Don’t exclusively follow others. Always do your own research.
* Luck counts too.
> * Conservative (30-70% band) War, Treaties & International Agreements, Economics, International Conflict, Elections
> * Moderate: (20-80% band) Markets, Interest Rates, Instability, Indicators, Health, Exchange Rates, Finance
> * Aggressive: (5%-95% band) Leaders, Weapons, Taxes, Legislation, International Organizations, Foreign Policy, Environment, Energy, Domestic Politics, Commodities
That's incredibly useful, thanks.