I assume that many of you have seen that Good Judgement is running a September Superforecasting Workshop. Although I would love to attend, the tickets are $1250 which is so far beyond my reach it's not even worth considering.
So what I wanted to put to the community was the idea of running a cheaper workshop or even a conference. There are some really amazing predicters in the community so I honestly think it would be possible to match the quality of the official workshop. I am happy to do the legwork in organizing it if people are interested? Apologies if this is not the place to put this.
On a related note, has anyone been lucky enough to attend the official workshop? If so, what was it like, did you feel it was worth it? Would love to hear your thoughts!
I'm looking at this in terms of available information.
Take GJO. For most intents and purposes, it already acts like a workshop. Especially if we take the time to read the past rationales of users who are now superforecasters, we can learn quite a bit ... for free. And we can of course learn a lot from the rationales of current top forecasters, which are at least that much valuable, if not more.
However, I am curious to see how such an initiative would turn out :)
@tabhuth@James-B I do concede that when I said, "would not remarkably change anything," this is incredibly vague and open to misinterpretation.
"Especially if we take the time to read the past rationales of users who are now superforecasters, we can learn quite a bit ... for free. And we can of course learn a lot from the rationales of current top forecasters, which are at least that much valuable, if not more."
>> Yes, I do agree very much. Even more, learn from the real world! Forecast on random things like what are the chances the next car you see will be red? Score yourself with Brier scoring. What's the chance you won't read everything on this post that's to come (i.e. retention rate?) What's the chance CNN's top headline is about Biden? Make a forecast. Score yourself. Create a calibration plot. Repeat: what's the chance that hurricane on the news will turn into a Category 4? Make a forecast. Score yourself. Receive feedback. What's the chance you'll encounter a red light when you approach an intersection? Observe. Take score. Learn. What's the chance the popular PC game Overwatch 2 receives a better Metacritic score than the original Overwatch PC game? Make your forecast, learn, adjust.
Then when you get comfortable with the short-term, start thinking in larger scopes: what's the chance there will be another Magnitude 9 earthquake in the next 20 years? Make a forecast. What's your 50% chance of a next mass casualty event in the United States with at least 50 deaths? What's your 95% probability? What's your 40% chance and 60% chance of when electric cars will comprise 50% or more of the cars for sale. Keep doing these exercises.
Think conditionally and long-term: IF China invades Taiwan, what's the probability America retaliates militarily? What's the chance that IF China and the US go to war, that China would win? Would China win but have more casualties? Would a nuclear weapon be used? What's the chance? IF humans cannot reverse global warming, what's the chance more than 7% of Florida goes below sea level in 50 years . . . in 100 years . . . in 200 years . . . in 3,000 years. If humans manage to colonize Mars, how much does it change the probability of the stability and functionality of the United Nations? Does it nudge a little up or down? By a lot? Or no change? Think counterfactually: Would the United Nations in its current structure have reduced the chance of WWII? Etc.
Anyways, yeah. There is a reason one of the best Superforecasters by the name of Bill Flack likes birdwatching. It's a very lonesome and humble activity. But it requires keen observation. A passion to observe and learn about the world. To see the scope of the world. You should be able to comfortably look at a tree and realize all the factors that influenced its growth. Compare it to another tree meters away. What's the chance it would've been tall as that tree had it grown 3 yards farther? 3 months sooner? Had there been a monsoon? Do tree survivability rates decrease in urban areas compared to rural areas? By how much? What is the biggest correlating factor in tree growth? Sunlight? Water? Soil nutrients?
Look at fish in a fish tank. What fish are most likely to get into fights? Why do you think so? How long would it take for algae to grow before I can perceive it with my eyes? How confident are you in that? What's your 90% confidence that when you feed your fish that all the food will be eaten in order to prevent wasted food and money?
Look at TMZ dot com. Observe the celebrity couples. What's the probability they'll still be together in 6-8 years? You learn that one of them has had 3 previous partners. How much does your probability shift then? What about the average couple? How much difference or deviation do you observe?
Estimate how many scientific publications may have 15% or more of its stated facts actually be false. Estimate how many lawyers there are in the United States. Compare the best estimates on Google with your best guess. How much were you off by? Why do you think so?
Learn how meteorologists forecast hurricanes, storms, rain, etc. Learn about ensemble forecasting. How to score spatial forecasts. How the CDC scores their epidemiological ensemble members. How to weigh different models together or throw some out all together.
TL;DR: The world is your best forum to learn how to become a better forecaster. From what I know, people are biased, corrupt, noisy, arrogant, overconfident, close-minded, tribal, selfish, and mischievous people out of touch with reality - to varying degrees. Be honest with yourself and the world.
Make the best, most logical decisions you can. That's good hygiene.
@James-B@tabhuth@404_NOT_FOUND I want to support the initiative and test the hypothesis that skills are better honed in a group of forecasters. Of course, each of you can do excellent work alone, but we'll never know how much better or worse the result will be in a group without trying.
What should we do right now to get it started - create a Telegram group and Discord channel? What's the point of being a super forecaster if you don't change the world? While even ordinary people are changing it just now, super forecasters should have a league that will take it to the next level.
"What's the point of being a super forecaster if you don't change the world? While even ordinary people are changing it just now, super forecasters should have a league that will take it to the next level."
>> While I agree in part, I think tonally it's a little hedgehog-y. I'm skeptical the world as a whole will accept forecasters who are consistently at the top of the crop and their value. Yes, in theory accurate forecasters should be weighted in the public commentary. However, it's like a market and the demand for it may be relatively low because the crowd at large doesn't care about probabilistic estimates to begin with. Perhaps in isolated and occasional moments would top forecasters have great success with the public, whatever that means. Further, there has been criticism that Supers failed to predict the invasion in Ukraine, Brexit, Trump's 2016 win, and even the first COVID-19 vaccine - topics of great interest to the public.
I would even argue against these criticisms though. Personally, having topped the leaderboard on the Superforecaster Workshops Challenge on my time here and reaching top 10 on the annual News challenge for 2 years straight, I performed better than average and most Supers on the Ukraine invasion, outperformed the crowd on the Taliban taking over Afghanistan, and performed slightly better than Supers on the first COVID vaccine emergence. Being in the top percentile doesn't mean we ALL miss big headlines. In addition, even though top forecasters missed some of the big ones, they are still reliably better over a longer period of time. Small gains in accuracy could yield thousands or even millions of dollars and lives that are saved in the aggregate if decision makers utilized Pro forecasts. For those criticizing how the Supers missed some of the biggest headlines of the decade, try beating them head-to-head on 100s of questions first. Who would you rather have helping you to make probabilistic decisions? The alternative is less accurate forecasters. Yes, Pro forecasts aren't foolproof. But it's some of the best we've measurably got.
12/2/2022 UPDATE: Think of pro forecasters as eyeglasses to see the world a little clearer.
@DR-R3W I'd also like to note that Pro forecasters already engage in highly active discussions and meet-ups with one another on their own times and for the companies they work for. Clients pay for Professional forecasts, too. Any side work would trade off time from this effort where people value forecasting enough that they are willing to pay for it. Where do the top forecasters go to get better? I doubt that more than 50% of them go to workshops to learn how to get better. I think it's almost an innate desire to get better, to grasp reality a little better. So again, the first and foremost place - and probably the most cheap way - to get lessons on how to become better is to just get on the bike and start peddling, as Superforecaster Warren Hatch says. Only with practice and constant feedback from the real world do you get better at something. It doesn't promise being elite in something, but it almost guarantees improvement, which the world would benefit from. @James-B
PS: To get better, start by probing for information. Asking questions. Especially about contradicting information. You want to keep adjusting your priors so that when the truth begins to emerge with great clarity, you're not caught off guard by being forced to make huge shifts in probability.
For instance, if you wanted to forecast if Musk would close his Twitter deal and acquire it by the end of the year, take note of his opinion. Then adjust based on his unreliability. Then adjust that talks are still going on. Adjust because shareholders promote the deal. Adjust because the Twitter CEO also wants the deal closed. Adjust based on trends from prediction markets. I think your current adjustment to 63% is a good adjustment!
@James-B@DR-R3W To be honest, I do find the idea interesting. If you do end up organising something like this, let me know. I would probably attend, out of pure curiosity. I live in NY state, btw.
Having said that, I do have differences of opinion, philosophically speaking. It's not so much that it's a bad idea, as much as an idea with potentially limited utility. (And I think this would be the same of the Superforecasting workshops as well.)
And something else that's holding me back is: a workshop would be fine in principle. However, what I would find really useful is not the how, why, what (tips and tricks, so to speak) but the when. This is an issue that I find while reading old comments. As far as I can see, the way to learn this is in real time.
Also, to re-iterate what I said, there are fantastic forecasters right here on GJO from whom we can learn a lot. Do remember: all Superforecasters start on GJO. It's people who are here now who will be Superforecasters in the future.
EDIT: I didn't realise that this comment was itself a forecast until I finished typing it, lol.
I do wish you the best in your organising endeavour. Do let me know if it transpires.
@probahilliby I love how you put forward arguments and dispute this - quite interesting to read.
I wanted to convey that I would like not only to predict on my own but also for all this to be used - forecasts for the sake of forecasts are not enough. Perhaps the problem is that I work for my own company and currently don't have a team to engage in highly active discussions and meet-ups lol Although I agree that decision-makers, and society as a whole, should listen more often to forecasts, even if they are not always accurate, to save lives at least. And I agree even more with those sources of knowledge, like the real world, that you listed - I use them, of course. I just strive to share experiences with those who pursue the same goals. I'm interested in evaluating problems not discussed on GJO that have shorter time frames and are verifiable. I think faster feedback from a network of forecasters would help significantly in skill development. Most GJO forecasters who make predictions don't justify the logic behind their conclusions because it goes beyond the platform.Communication and sharing of the decision-making process are vital to me. Maybe I would even like to blog on this topic, but for this, it is necessary to test hypotheses in real life, while alone it will be one-sided and slow testing. In addition, some forecasters have skills in big data, AI development and other skills that can be useful in the work of a team of forecasters. I even can say that my goal is to create a team and work out an algorithm.I have a lot of notes and observations that I cannot physically verify, but perhaps in a group of people, we could take a critical look and pass it through the prism of collective knowledge. At large, it's all about practice and the speed of feedback from a network of forecasters. Because as @tabhuth has mentioned, timeliness is critical.
@DR-R3W That's a fair assessment. If @James-B does kickstart something of that sort, I'm open to participating and don't refute the positive influence it would have in this world.
If you do have a blog, I'm very curious to read it, Dr.!
Somewhat related: I've long thought it would be great if there were a forecasting-related subreddit....a place where people could talk through both high-level theory and best practices, as well as get into the weeds on specific forecasts. Inspired by this thread, I went ahead and created one.
I will post to the Superforecasters forum and see if anyone inspired to jump in from that side of things. I suspect it could become a pretty active community within a short period of time. If anyone is interested, feel free to join and post...
So what I wanted to put to the community was the idea of running a cheaper workshop or even a conference. There are some really amazing predicters in the community so I honestly think it would be possible to match the quality of the official workshop. I am happy to do the legwork in organizing it if people are interested? Apologies if this is not the place to put this.
On a related note, has anyone been lucky enough to attend the official workshop? If so, what was it like, did you feel it was worth it? Would love to hear your thoughts!
@probahilliby
You are right that "accuracy can be improved with teamwork and shared information".
You are probably right that it is "likely to attract those who think they know more than average / know it all".
You may be right that "a space like you describe would not remarkably change anything".
---
@James-B
I'm looking at this in terms of available information.
Take GJO. For most intents and purposes, it already acts like a workshop. Especially if we take the time to read the past rationales of users who are now superforecasters, we can learn quite a bit ... for free. And we can of course learn a lot from the rationales of current top forecasters, which are at least that much valuable, if not more.
However, I am curious to see how such an initiative would turn out :)
@tabhuth @James-B I do concede that when I said, "would not remarkably change anything," this is incredibly vague and open to misinterpretation.
"Especially if we take the time to read the past rationales of users who are now superforecasters, we can learn quite a bit ... for free. And we can of course learn a lot from the rationales of current top forecasters, which are at least that much valuable, if not more."
>> Yes, I do agree very much. Even more, learn from the real world! Forecast on random things like what are the chances the next car you see will be red? Score yourself with Brier scoring. What's the chance you won't read everything on this post that's to come (i.e. retention rate?) What's the chance CNN's top headline is about Biden? Make a forecast. Score yourself. Create a calibration plot. Repeat: what's the chance that hurricane on the news will turn into a Category 4? Make a forecast. Score yourself. Receive feedback. What's the chance you'll encounter a red light when you approach an intersection? Observe. Take score. Learn. What's the chance the popular PC game Overwatch 2 receives a better Metacritic score than the original Overwatch PC game? Make your forecast, learn, adjust.
Then when you get comfortable with the short-term, start thinking in larger scopes: what's the chance there will be another Magnitude 9 earthquake in the next 20 years? Make a forecast. What's your 50% chance of a next mass casualty event in the United States with at least 50 deaths? What's your 95% probability? What's your 40% chance and 60% chance of when electric cars will comprise 50% or more of the cars for sale. Keep doing these exercises.
Think conditionally and long-term: IF China invades Taiwan, what's the probability America retaliates militarily? What's the chance that IF China and the US go to war, that China would win? Would China win but have more casualties? Would a nuclear weapon be used? What's the chance? IF humans cannot reverse global warming, what's the chance more than 7% of Florida goes below sea level in 50 years . . . in 100 years . . . in 200 years . . . in 3,000 years. If humans manage to colonize Mars, how much does it change the probability of the stability and functionality of the United Nations? Does it nudge a little up or down? By a lot? Or no change? Think counterfactually: Would the United Nations in its current structure have reduced the chance of WWII? Etc.
Anyways, yeah. There is a reason one of the best Superforecasters by the name of Bill Flack likes birdwatching. It's a very lonesome and humble activity. But it requires keen observation. A passion to observe and learn about the world. To see the scope of the world. You should be able to comfortably look at a tree and realize all the factors that influenced its growth. Compare it to another tree meters away. What's the chance it would've been tall as that tree had it grown 3 yards farther? 3 months sooner? Had there been a monsoon? Do tree survivability rates decrease in urban areas compared to rural areas? By how much? What is the biggest correlating factor in tree growth? Sunlight? Water? Soil nutrients?
Look at fish in a fish tank. What fish are most likely to get into fights? Why do you think so? How long would it take for algae to grow before I can perceive it with my eyes? How confident are you in that? What's your 90% confidence that when you feed your fish that all the food will be eaten in order to prevent wasted food and money?
Look at TMZ dot com. Observe the celebrity couples. What's the probability they'll still be together in 6-8 years? You learn that one of them has had 3 previous partners. How much does your probability shift then? What about the average couple? How much difference or deviation do you observe?
Estimate how many scientific publications may have 15% or more of its stated facts actually be false. Estimate how many lawyers there are in the United States. Compare the best estimates on Google with your best guess. How much were you off by? Why do you think so?
Learn how meteorologists forecast hurricanes, storms, rain, etc. Learn about ensemble forecasting. How to score spatial forecasts. How the CDC scores their epidemiological ensemble members. How to weigh different models together or throw some out all together.
TL;DR: The world is your best forum to learn how to become a better forecaster. From what I know, people are biased, corrupt, noisy, arrogant, overconfident, close-minded, tribal, selfish, and mischievous people out of touch with reality - to varying degrees. Be honest with yourself and the world.
Make the best, most logical decisions you can. That's good hygiene.
What should we do right now to get it started - create a Telegram group and Discord channel?
What's the point of being a super forecaster if you don't change the world? While even ordinary people are changing it just now, super forecasters should have a league that will take it to the next level.
@DR-R3W
"What's the point of being a super forecaster if you don't change the world? While even ordinary people are changing it just now, super forecasters should have a league that will take it to the next level."
>> While I agree in part, I think tonally it's a little hedgehog-y. I'm skeptical the world as a whole will accept forecasters who are consistently at the top of the crop and their value. Yes, in theory accurate forecasters should be weighted in the public commentary. However, it's like a market and the demand for it may be relatively low because the crowd at large doesn't care about probabilistic estimates to begin with. Perhaps in isolated and occasional moments would top forecasters have great success with the public, whatever that means. Further, there has been criticism that Supers failed to predict the invasion in Ukraine, Brexit, Trump's 2016 win, and even the first COVID-19 vaccine - topics of great interest to the public.
I would even argue against these criticisms though. Personally, having topped the leaderboard on the Superforecaster Workshops Challenge on my time here and reaching top 10 on the annual News challenge for 2 years straight, I performed better than average and most Supers on the Ukraine invasion, outperformed the crowd on the Taliban taking over Afghanistan, and performed slightly better than Supers on the first COVID vaccine emergence. Being in the top percentile doesn't mean we ALL miss big headlines. In addition, even though top forecasters missed some of the big ones, they are still reliably better over a longer period of time. Small gains in accuracy could yield thousands or even millions of dollars and lives that are saved in the aggregate if decision makers utilized Pro forecasts. For those criticizing how the Supers missed some of the biggest headlines of the decade, try beating them head-to-head on 100s of questions first. Who would you rather have helping you to make probabilistic decisions? The alternative is less accurate forecasters. Yes, Pro forecasts aren't foolproof. But it's some of the best we've measurably got.
12/2/2022 UPDATE: Think of pro forecasters as eyeglasses to see the world a little clearer.
@DR-R3W I'd also like to note that Pro forecasters already engage in highly active discussions and meet-ups with one another on their own times and for the companies they work for. Clients pay for Professional forecasts, too. Any side work would trade off time from this effort where people value forecasting enough that they are willing to pay for it. Where do the top forecasters go to get better? I doubt that more than 50% of them go to workshops to learn how to get better. I think it's almost an innate desire to get better, to grasp reality a little better. So again, the first and foremost place - and probably the most cheap way - to get lessons on how to become better is to just get on the bike and start peddling, as Superforecaster Warren Hatch says. Only with practice and constant feedback from the real world do you get better at something. It doesn't promise being elite in something, but it almost guarantees improvement, which the world would benefit from. @James-B
PS: To get better, start by probing for information. Asking questions. Especially about contradicting information. You want to keep adjusting your priors so that when the truth begins to emerge with great clarity, you're not caught off guard by being forced to make huge shifts in probability.
For instance, if you wanted to forecast if Musk would close his Twitter deal and acquire it by the end of the year, take note of his opinion. Then adjust based on his unreliability. Then adjust that talks are still going on. Adjust because shareholders promote the deal. Adjust because the Twitter CEO also wants the deal closed. Adjust based on trends from prediction markets. I think your current adjustment to 63% is a good adjustment!
@James-B @DR-R3W To be honest, I do find the idea interesting. If you do end up organising something like this, let me know. I would probably attend, out of pure curiosity. I live in NY state, btw.
Having said that, I do have differences of opinion, philosophically speaking. It's not so much that it's a bad idea, as much as an idea with potentially limited utility. (And I think this would be the same of the Superforecasting workshops as well.)
For a start, I'm not sure if any workshop would be more useful than this: https://www.gjopen.com/comments/1165014
And something else that's holding me back is: a workshop would be fine in principle. However, what I would find really useful is not the how, why, what (tips and tricks, so to speak) but the when. This is an issue that I find while reading old comments. As far as I can see, the way to learn this is in real time.
Also, to re-iterate what I said, there are fantastic forecasters right here on GJO from whom we can learn a lot. Do remember: all Superforecasters start on GJO. It's people who are here now who will be Superforecasters in the future.
EDIT: I didn't realise that this comment was itself a forecast until I finished typing it, lol.
I do wish you the best in your organising endeavour. Do let me know if it transpires.
Although I agree that decision-makers, and society as a whole, should listen more often to forecasts, even if they are not always accurate, to save lives at least. And I agree even more with those sources of knowledge, like the real world, that you listed - I use them, of course. I just strive to share experiences with those who pursue the same goals. I'm interested in evaluating problems not discussed on GJO that have shorter time frames and are verifiable. I think faster feedback from a network of forecasters would help significantly in skill development.
Most GJO forecasters who make predictions don't justify the logic behind their conclusions because it goes beyond the platform.Communication and sharing of the decision-making process are vital to me. Maybe I would even like to blog on this topic, but for this, it is necessary to test hypotheses in real life, while alone it will be one-sided and slow testing. In addition, some forecasters have skills in big data, AI development and other skills that can be useful in the work of a team of forecasters. I even can say that my goal is to create a team and work out an algorithm.I have a lot of notes and observations that I cannot physically verify, but perhaps in a group of people, we could take a critical look and pass it through the prism of collective knowledge. At large, it's all about practice and the speed of feedback from a network of forecasters. Because as @tabhuth has mentioned, timeliness is critical.
@DR-R3W That's a fair assessment. If @James-B does kickstart something of that sort, I'm open to participating and don't refute the positive influence it would have in this world.
If you do have a blog, I'm very curious to read it, Dr.!
Interesting and worthwhile discussion here.
Somewhat related: I've long thought it would be great if there were a forecasting-related subreddit....a place where people could talk through both high-level theory and best practices, as well as get into the weeds on specific forecasts. Inspired by this thread, I went ahead and created one.
https://www.reddit.com/r/forecasters/
I will post to the Superforecasters forum and see if anyone inspired to jump in from that side of things. I suspect it could become a pretty active community within a short period of time. If anyone is interested, feel free to join and post...