I made a $500 bet at even odds that Lee Sedol will win the match against AlphaGo. If I win, I intend to donate my winnings to one of GiveWell's top charities (probably the Against Malaria Foundation).
At the time of the bet I would have assigned a ~30% probability estimate to AlphaGo winning, however I have since learned that AlphaGo was trained on amateur games rather than professional games (6d-9d games rather than 6p-9p games). Learning this raises my estimate for two reasons: (1) there is a possibility that Google will train AlphaGo on higher-level games (professional games) before the March match and (2) even if AlphaGo is not trained on higher level games, the fact that AlphaGo played at the level it did in October despite being trained on these lower level games shows that it is capable of learning and improving more by playing itself than I thought it could, so there is more potential for it to learn enough to reach or surpass Lee Sedol's level.
I note that @Dima-K updated their prediction from 5% to 10% upon learning this: "Going up because I just realized that a training set is likely to be based on pro games, not on the amateur online database they used earlier." I think it warrants a greater update, hence my 30% to 40% update.
Now, backtracking, I couldn't find any good base rates to start my prediction at. I noticed that a confident consensus seemed to be that October-AlphaGo was much weaker than Lee Sedol, so the question "How much can AlphaGo improve by March?" is really the main factor that affects my probability estimate.
After learning more about the difference in skill level between Fan Hui and Lee Sedol, it initially seemed likely to me that AlphaGo would improve by some amount, closer to Lee Sedol's level, but wouldn't be able to improve by the rather-large-seeming amount necessary to reach or surpass Lee Sedol's level.
Reading some of the rationales that people wrote here, there were a few things I noticed:
(1) Those who wrote long rationales (which I assumed correlated with how much time they spent thinking about this, and gathering information) tended to have lower forecasts.
(3) Most of those who made forecasts of >60% seemed to only mentioned reasons to justify their high probability estimate without mentioning counter-arguments (confirmation bias).
For example, @Abusawyer said "Google wouldn't agree to the deal if it was not reasonably confident of winning" and predicted 75%. Other plausible reasons why Google might offer / agree to the deal: (1) Test to see how good AlphaGo actually is / better determine AlphaGo's playing strength. (2) Learn something else from the match against the strong Lee Sedol (2) They may have decided to challenge him earlier near October as soon as they saw that AlphaGo was stronger than Lee Sedol. Why wait more than 5 months to try against the next top professional? Even if they weren't confident that they would win against Sedol, it still seems plausible to me that they would challenge him because it's the next logical step. IBM challenged Kasparov before Deep Blue was strong enough to defeat him in the match. Google may have even less pressure than IBM had to do well. Although I note that @Abusawyer has a great Brier score: 0.072 (median: 0.203). Also see: https://www.gjopen.com/comments/comments/118626
I also note that the consensus probability has been going down over time. I wonder if this is due to the naive view that "AlphaGo won 5-0 in October, so it will probably win in March too" being more prominent initially.
Finally, my disagreement with those informed forecasters who are predicting ~10% just stems from general uncertainty. I just don't feel that I can be that confident. Although, at the same time I am not confident that I shouldn't be assigning a probability that low.
I am not a Go player. I played a couple dozen blitz games online once after learning the rules a couple years ago on KGS Server, but don't think I ever got below ~15-20k.
It's funny, because the initial reaction from pro circles is that, like Myungwan Kim, they believe it's impossible for AlphaGo to beat Lee Sedol But after some time has past, and they became better informed/rational, some are starting to think that maybe AlphaGo is stronger/has more potential than they intially gave it credit for.
The difference between Lee Sedol and Fan Hui is absolutely huge, in that Fan has almost no chance to beat Lee Sedol, that much is fact. However, like I said in my first prediction, the absolute difference isn't that big. It's kind of like 100m olympic sprinters, maybe the no.10 person can never outrun the no.1, but the absolute difference in time is tiny.
I can see AlphaGo easily overpowering Lee Sedol, if it doesn't have the same kind of bottleneck humans do. But if it does, then it likely won't win.
Oh, and I think there's an over obsession with links/sources here. If people want some, I do have some, but they're all in Chinese.
Thanks for the comment. It's indeed interesting that the change in professional view (I largely take your word for it) is trending in the opposite direction as the GJ consensus.
re: your second point: Intuitively, Go strikes me as a game where Lee Sedol is much further from "best play" than Magnus Carlsen (#1 chess player) is from best play in chess. So while the best chess computer is better than Carlsen, Carlsen can still draw sometimes when he plays his best game without any inaccuracies over the course of the game. On the other hand, I'd guess that *every* game that Sedol plays has inaccuracies, simply due to the complexity of the game. I would thus expect future Go computers that approach best play to capitalize on this and win against Sedol very close to 99-100% of the time.
So I agree that it's possible that a computer could far overshoot Sedol's skill level, and that the question is how likely is it that AlphaGo's improvements will be small or large.
(I removed most of my links, because I agree they are unnecessary.)
@WJK I didn't mean your links were unnecessary, it's just that some of the other commentators refuse to accept commonly accepted things without a "link" (despite the fact that I could really just write the same thing somewhere else and link it).
That's the only source I have in English currently, it's an interview with Kang Dongyun, the only person who won an international tournament last year, other than Ke Jie (I'm pretty sure).
Quote: "AlphaGo is currently a hot issue, what do you think about the upcoming match against Lee Sedol 9p?
Kang: I saw the games of AlphaGo, and it played really well. However, it made some big mistakes in joseki. I don’t understand how computers can make such big mistakes. Most people seem to think that Lee will win easily, but I’d be scared if I were playing against AlphaGo."
"I heard that Kim Jiseok 9p said that AlphaGo won’t lose that easily against Lee Sedol 9p.
Kang: Oh, he has the same opinion as me! He’s very learned… [laughs]"
Yeah, I look at some of the professional commentaries (all in chinese, unfortunately), and in every single game even in the finals of very important international tournaments, both players makes a huge number of mistakes.
I made a $500 bet at even odds that Lee Sedol will win the match against AlphaGo. If I win, I intend to donate my winnings to one of GiveWell's top charities (probably the Against Malaria Foundation).
At the time of the bet I would have assigned a ~30% probability estimate to AlphaGo winning, however I have since learned that AlphaGo was trained on amateur games rather than professional games (6d-9d games rather than 6p-9p games). Learning this raises my estimate for two reasons: (1) there is a possibility that Google will train AlphaGo on higher-level games (professional games) before the March match and (2) even if AlphaGo is not trained on higher level games, the fact that AlphaGo played at the level it did in October despite being trained on these lower level games shows that it is capable of learning and improving more by playing itself than I thought it could, so there is more potential for it to learn enough to reach or surpass Lee Sedol's level.
I note that @Dima-K updated their prediction from 5% to 10% upon learning this: "Going up because I just realized that a training set is likely to be based on pro games, not on the amateur online database they used earlier." I think it warrants a greater update, hence my 30% to 40% update.
Now, backtracking, I couldn't find any good base rates to start my prediction at. I noticed that a confident consensus seemed to be that October-AlphaGo was much weaker than Lee Sedol, so the question "How much can AlphaGo improve by March?" is really the main factor that affects my probability estimate.
After learning more about the difference in skill level between Fan Hui and Lee Sedol, it initially seemed likely to me that AlphaGo would improve by some amount, closer to Lee Sedol's level, but wouldn't be able to improve by the rather-large-seeming amount necessary to reach or surpass Lee Sedol's level.
Reading some of the rationales that people wrote here, there were a few things I noticed:
(1) Those who wrote long rationales (which I assumed correlated with how much time they spent thinking about this, and gathering information) tended to have lower forecasts.
(2) The few "superforecasters" I stumbled into all had <50% forecasts, some much lower than 50% (Update: @einsteinjs 8% https://www.gjopen.com/comments/comments/123061), @Jean-Pierre 10% https://www.gjopen.com/comments/comments/118530 and @GJDrew 25% https://www.gjopen.com/comments/comments/122608).
(3) Most of those who made forecasts of >60% seemed to only mentioned reasons to justify their high probability estimate without mentioning counter-arguments (confirmation bias).
For example, @Abusawyer said "Google wouldn't agree to the deal if it was not reasonably confident of winning" and predicted 75%. Other plausible reasons why Google might offer / agree to the deal: (1) Test to see how good AlphaGo actually is / better determine AlphaGo's playing strength. (2) Learn something else from the match against the strong Lee Sedol (2) They may have decided to challenge him earlier near October as soon as they saw that AlphaGo was stronger than Lee Sedol. Why wait more than 5 months to try against the next top professional? Even if they weren't confident that they would win against Sedol, it still seems plausible to me that they would challenge him because it's the next logical step. IBM challenged Kasparov before Deep Blue was strong enough to defeat him in the match. Google may have even less pressure than IBM had to do well. Although I note that @Abusawyer has a great Brier score: 0.072 (median: 0.203). Also see: https://www.gjopen.com/comments/comments/118626
I also note that the consensus probability has been going down over time. I wonder if this is due to the naive view that "AlphaGo won 5-0 in October, so it will probably win in March too" being more prominent initially.
Finally, my disagreement with those informed forecasters who are predicting ~10% just stems from general uncertainty. I just don't feel that I can be that confident. Although, at the same time I am not confident that I shouldn't be assigning a probability that low.
I am not a Go player. I played a couple dozen blitz games online once after learning the rules a couple years ago on KGS Server, but don't think I ever got below ~15-20k.
A thread I made on the Go subreddit asking: "How likely is AlphaGo to win 5-0 against Lee Se-dol in March?" https://www.reddit.com/r/baduk/comments/43xret/how_likely_is_alphago_to_win_50_against_lee_sedol/
It's funny, because the initial reaction from pro circles is that, like Myungwan Kim, they believe it's impossible for AlphaGo to beat Lee Sedol But after some time has past, and they became better informed/rational, some are starting to think that maybe AlphaGo is stronger/has more potential than they intially gave it credit for.
The difference between Lee Sedol and Fan Hui is absolutely huge, in that Fan has almost no chance to beat Lee Sedol, that much is fact. However, like I said in my first prediction, the absolute difference isn't that big. It's kind of like 100m olympic sprinters, maybe the no.10 person can never outrun the no.1, but the absolute difference in time is tiny.
I can see AlphaGo easily overpowering Lee Sedol, if it doesn't have the same kind of bottleneck humans do. But if it does, then it likely won't win.
Oh, and I think there's an over obsession with links/sources here. If people want some, I do have some, but they're all in Chinese.
Thanks for the comment. It's indeed interesting that the change in professional view (I largely take your word for it) is trending in the opposite direction as the GJ consensus.
re: your second point: Intuitively, Go strikes me as a game where Lee Sedol is much further from "best play" than Magnus Carlsen (#1 chess player) is from best play in chess. So while the best chess computer is better than Carlsen, Carlsen can still draw sometimes when he plays his best game without any inaccuracies over the course of the game. On the other hand, I'd guess that *every* game that Sedol plays has inaccuracies, simply due to the complexity of the game. I would thus expect future Go computers that approach best play to capitalize on this and win against Sedol very close to 99-100% of the time.
So I agree that it's possible that a computer could far overshoot Sedol's skill level, and that the question is how likely is it that AlphaGo's improvements will be small or large.
(I removed most of my links, because I agree they are unnecessary.)
@WJK I didn't mean your links were unnecessary, it's just that some of the other commentators refuse to accept commonly accepted things without a "link" (despite the fact that I could really just write the same thing somewhere else and link it).
https://gogameguru.com/kang-dongyun-wins-the-20th-lg-cup/
That's the only source I have in English currently, it's an interview with Kang Dongyun, the only person who won an international tournament last year, other than Ke Jie (I'm pretty sure).
Quote: "AlphaGo is currently a hot issue, what do you think about the upcoming match against Lee Sedol 9p?
Kang: I saw the games of AlphaGo, and it played really well. However, it made some big mistakes in joseki. I don’t understand how computers can make such big mistakes. Most people seem to think that Lee will win easily, but I’d be scared if I were playing against AlphaGo."
"I heard that Kim Jiseok 9p said that AlphaGo won’t lose that easily against Lee Sedol 9p.
Kang: Oh, he has the same opinion as me! He’s very learned… [laughs]"
Yeah, I look at some of the professional commentaries (all in chinese, unfortunately), and in every single game even in the finals of very important international tournaments, both players makes a huge number of mistakes.