I assume that many of you have seen that Good Judgement is running a September Superforecasting Workshop. Although I would love to attend, the tickets are $1250 which is so far beyond my reach it's not even worth considering.
So what I wanted to put to the community was the idea of running a cheaper workshop or even a conference. There are some really amazing predicters in the community so I honestly think it would be possible to match the quality of the official workshop. I am happy to do the legwork in organizing it if people are interested? Apologies if this is not the place to put this.
On a related note, has anyone been lucky enough to attend the official workshop? If so, what was it like, did you feel it was worth it? Would love to hear your thoughts!
@probahilliby I don't remember exactly. Better than it is today. I think maybe my Brier was down in the .17-18 realm versus a median of .24-.26 or something. I slacked off earlier this year and bombed a few long-dated questions that I stopped updating. I was heavily focused on the Coronavirus Challenge and did really well there...for whatever reason I seemed to over-perform on that subject matter. Frankly, I may have lucked out because I generally used a "it's going to be worse than everyone thinks it is" heuristic from the early days...and that happened to be the right take most of the time for a very, very long time. :-)
Unless I am missing something the ideas under discussion are: 1. Discord about forecasting, which has already been created by @james-b (I joined this morning). 2. Some kind of Superforecasting workshop (for example: special for GJO users; seats in regular workshop; and/or for new Superforecasters). 3. Have regular presentations on forecasting, forecasting research, from subject-matter experts, challenge sponsors, Superforecasters as a whole. 4. Reddit AMA with a Superforecaster (I should be able to do that personally). 5. Developing a “network of Superforecasters who can share their practices”, “Community for improvement”. This is something I personally know as a “community of practice”. 6. Create new "more off-topic" threads on GJO, but still somewhat related to forecasting. Anything else? Happy to jump on a call on these soon, or any other ideas, if only to get to know each other better. We have had a few similar calls with previous cohorts of forecasters and they have been understood to be helpful.
@LuisEnrique Thanks for your detailed consideration. I think the key one is the Superforecasting workshop (point #2) and the ideas discussed about having bias histograms, calibration plots, and forecaster similarity data visualizations which you can find in the Discord under #ideas.
My particular interest is point #3 as I feel there is somewhat of a disconnect between sponsors/partners (like the various Universities and The Economist) and GJO forecasters. Perhaps there could be prizes, occasional references on their websites on key forecast rationales, trends, or top forecasters on the respective challenge leaderboard. A good start was the UBS Asset Management Investments Recruitment Challenge where top performers meeting the recruitment criteria would be included in the UBS program.
Point #5 is interesting to me too. This video {https://www.youtube.com/watch?v=6POQjSjIXWk} was a great resource for sharing improvement strategies from those who most succeeded on GJP/GJO. Future versions need not be in video format, but the point about having a "community for improvement" is a great idea.
With point #6, I think it's okay but it needs to be ironed out what is considered off topic and what is deemed acceptable. I anticipate more moderation would be needed for this kind of Open thread in the long run.
@LuisEnrique I would add that I am about to start work on a project to provide statistics and analytics for forecasters such as calibration curves, similarity scores to other forecasters, performance across topics, histograms of initial forecasts to showcase bias, etc. The discussion is in the #ideas channel on the discord. If anyone else with programming experience (especially frontend) is interested in helping me with this I would be grateful!
So what I wanted to put to the community was the idea of running a cheaper workshop or even a conference. There are some really amazing predicters in the community so I honestly think it would be possible to match the quality of the official workshop. I am happy to do the legwork in organizing it if people are interested? Apologies if this is not the place to put this.
On a related note, has anyone been lucky enough to attend the official workshop? If so, what was it like, did you feel it was worth it? Would love to hear your thoughts!
@lbiii I'm sorry to say that heuristic would fail you on Monkeypox for the next several weeks :)
Jokes aside, I'm glad to see your success in that particular challenge and with those impressive Brier / Relative Score.
Any advice? Good memories?
PS: Primarily why did you stop updating?
@James-B @DR-R3W @404_NOT_FOUND @probahilliby @tabhuth @Chrisp @lbiii @Syzygy3 (let me know if I forgot to tag anyone)
Unless I am missing something the ideas under discussion are:1. Discord about forecasting, which has already been created by @james-b (I joined this morning).
2. Some kind of Superforecasting workshop (for example: special for GJO users; seats in regular workshop; and/or for new Superforecasters).
3. Have regular presentations on forecasting, forecasting research, from subject-matter experts, challenge sponsors, Superforecasters as a whole.
4. Reddit AMA with a Superforecaster (I should be able to do that personally).
5. Developing a “network of Superforecasters who can share their practices”, “Community for improvement”. This is something I personally know as a “community of practice”.
6. Create new "more off-topic" threads on GJO, but still somewhat related to forecasting.
Anything else? Happy to jump on a call on these soon, or any other ideas, if only to get to know each other better. We have had a few similar calls with previous cohorts of forecasters and they have been understood to be helpful.
For context on @probahilliby's post just now on the gasoline question, please see above:
@jleibowich @MrLittleTexas @belikewater @Tolga @Ryan-Beck @pstamp @LunaSol @pietrokc @Anneinak @JonathanMann @yagudin @LokiOdinevich @cmeinel
@LuisEnrique Thanks for your detailed consideration. I think the key one is the Superforecasting workshop (point #2) and the ideas discussed about having bias histograms, calibration plots, and forecaster similarity data visualizations which you can find in the Discord under #ideas.
My particular interest is point #3 as I feel there is somewhat of a disconnect between sponsors/partners (like the various Universities and The Economist) and GJO forecasters. Perhaps there could be prizes, occasional references on their websites on key forecast rationales, trends, or top forecasters on the respective challenge leaderboard. A good start was the UBS Asset Management Investments Recruitment Challenge where top performers meeting the recruitment criteria would be included in the UBS program.
Point #5 is interesting to me too. This video {https://www.youtube.com/watch?v=6POQjSjIXWk} was a great resource for sharing improvement strategies from those who most succeeded on GJP/GJO. Future versions need not be in video format, but the point about having a "community for improvement" is a great idea.
With point #6, I think it's okay but it needs to be ironed out what is considered off topic and what is deemed acceptable. I anticipate more moderation would be needed for this kind of Open thread in the long run.
Swinging back to point #2, this is exciting.
See also for context: https://www.gjopen.com/comments/1524308
@LuisEnrique I would add that I am about to start work on a project to provide statistics and analytics for forecasters such as calibration curves, similarity scores to other forecasters, performance across topics, histograms of initial forecasts to showcase bias, etc. The discussion is in the #ideas channel on the discord. If anyone else with programming experience (especially frontend) is interested in helping me with this I would be grateful!
@probahilliby
Yes, we should be able to make something work for #2.
On #3, different challenges have had different models. But yes, I see the point you are making.
#5 is an idea I like very much, and we have explored it in different ways. I would personally like to revisit it soon.
What topics would be of interest for #6?
Again, more than happy to set up a call for these or other topics.
@james-b
Very interesting. I will ping you.