There's no comparison between the World Champion (9-dan pro) and the European Champion (2-dan pro). That would be like comparing World Chess Champion Magnus Carlsen to an ordinary International Master. The Europeans have a long way to go before they play Go as well as the Koreans, Chinese, and Japanese. Maybe AlphaGo could, in this analogy, be compared to a weak Grandmaster but I am not certain of this. Fan Hui didn't seem to be taking the match seriously.
The wildcard is how much AlphaGo improves in the time between the two matches and whether Lee Sedol will have access to any of its games. Regarding the former, my guess is not nearly enough to win the match, let alone a single game.
@GJDrew, the only reason why I'm at 10% and not 1% is because I doubt Lee Sedol will be given any of AlphaGo's pre-match games. If my hunch is correct, this adds some element of uncertainty.
@GLDrew That's the only thing that worries me -that Lee Sedol won't take the match seriously. I don't know anything about Go, but people have said that Fan Hui was playing an unconventional strategy and not trying very hard to win. I watched the week-long competition between the University of Pittsburgh's computer "Claudico" and four elite online poker pros last summer. Heads-up No-Limit Hold'em. The poker pros won by an average of 12 bb, which was judged not statistically significant (though the win rates of a couple of them toward the end of the competition were considerably higher). The bottom line was that the pros could beat Claudico, but two of them seemed to be trying and two did not. Even the two who were trying could have done better by comparing notes. I learned that people don't always take these things seriously, even though a lot of people are watching and rooting for them. And I don't know anything about Lee Sedol.
I watched the same poker match! Some of the poker pros were definitely trying to figure out the weird game of computer (for their own later use) rather than trying to necessarily maximize win chances. Especially in poker, computers often discover new ways to play that are different than conventional wisdom and pros might see it as a good chance to explore.
@Jean-Pierre How confident are you in this AlphaGo prediction? Would you be willing to wager money in a bet? If so, at what odds?
I am trying to figure out how much I can learn from your very low ~10% prediction given your superforecaster status. You are clearly great at forecasting, but maybe in this particular case you didn't think as carefully about it as some of the other people estimating 60+% that AlphaGo will win (?), so I'm not sure how much I should believe your low estimate and lower mine from what it otherwise would be. Thanks.
I'm very confident in this forecast unlike, say, my Super Bowl forecast. I have thought about this one quite a lot. I just haven't posted my thoughts here (too time-consuming). Of the 50 or so questions (!) that I'm forecasting right now, I would rank this among my top 5 in terms of confidence-level. Would I wager money? If I were a betting man, I would.
My forecast is actually 5% now because Lee Sedol (9-dan pro) in a different league compared to Fan Hui (2-dan pro). Fan Hui would be very lucky to win 1 in 10 even games against Lee Sedol. And Fan Hui won 2 against AlphaGo. My sense is he under-performed and could have won more. Lee is also far better than the ordinary 9-dan pro and has a chance of becoming one of the greats. A weak 9-dan pro might be able to win 1 out of 5 against Lee. But, then again, maybe not. Although AlphaGo will improve before the match, I seriously doubt it will be playing at Lee's level any time soon. I would even give high odds (65%-75%) of Lee winning 5-0.
@WJK, don't change your forecast because of mine. My Brier scores have been awful (compared to the consensus) since the New Year. Frankly, I'm surprised I'm still on the GJP Classic Geopolitics Leaderboard.
@Jean-Pierre, how closely have you examined the question of the likelihood that AlphaGo will improve enough to reach Lee Sedol's level by March? I see this is the critical question. Yes, it's definitely quite a large gap to close, but AlphaGo is already a large step up from past Go programs, and the paper they published said they had several ways to improve. What makes you so sure that the improvements won't be large enough to close the gap?
Edit: Also what is your most recent forecast? Is it still 10%?
I'm skeptical of the idea that five or six months is enough time to close the gap. For one, it seems to me that AlphaGo still plays very much like a computer. For example, Myungwan Kim argues that AlphaGo has trouble timing moves and assessing the changing value of stones. These weaknesses are deadly. They are also typical of Go computers. And the persistence of these weaknesses suggests to me that Google, in spite of their accomplishments, still hasn't solved the main problem with Go-playing computers. Go is a game with lots of ambiguity. Much more than chess. Based on what I've seen so far, I'm convinced AlphaGo is a lot better when brute force is called for rather than intuitive reasoning.
Myungwan Kim also argues that AlphaGo really needs to play more games against 9-dan pros in order to improve to their level. If it keeps playing mostly against itself, its playing strength will plateau. I have no reason to think he's wrong on that point. His commentary is in the link I provided. I recommend watching it.
Thanks the response. I've watched the Myungwan Kim commentary video. I wonder though, if AlphaGo really was expected to still be significantly below Sedol's strength (such that his match-winning chances were only 5%) in March, does it really make sense that the DeepMind team would have issued the challenge date of the match for March upon publishing the paper in late January? Why not schedule it for April or May to give themselves more time to improve so that they actually have a chance in the match? It thus seems to me that the seemingly-early challenge date is evidence that they are actually stronger than you say.
Also, even if they wanted to challenge him soon in March just to get better feedback of their strength and publicize their success so far, would they really offer a $1 million prize if they thought their odds were only 5%?
So even if I were to agree with you that 5% is a good prior for what I would have assigned in October for the probability that AlphaGo would win a match in March against Sedol, it seems that the fact that upon publication of their paper in late January they announced the match just over a month away in March (as opposed to announcing it for April/May), that one should update this 5% prior significantly to make it higher. What are reasons for keeping it low (<25%)?
I don't want to read too much into the DeepMind team's actions.
My prior would be closer to 1%. My current range of 5%-10% takes into account improvements in AlphaGo's level of play since October. Of course, I don't know what its level of play will be come match time and I could be making a serious mistake. But Kim's arguments are sound. In the final analysis, it took 9 years for chess computers to go from beating an International Master for the first time to beating world champion Garry Kasparov. Maybe go computers will make the same leap in a much shorter timeframe. But five months is really ambitious.
Thanks for the comments and questions. I'll consider what you say.
There's no comparison between the World Champion (9-dan pro) and the European Champion (2-dan pro). That would be like comparing World Chess Champion Magnus Carlsen to an ordinary International Master. The Europeans have a long way to go before they play Go as well as the Koreans, Chinese, and Japanese. Maybe AlphaGo could, in this analogy, be compared to a weak Grandmaster but I am not certain of this. Fan Hui didn't seem to be taking the match seriously.
The wildcard is how much AlphaGo improves in the time between the two matches and whether Lee Sedol will have access to any of its games. Regarding the former, my guess is not nearly enough to win the match, let alone a single game.
Interesting observation: "Fan Hui didn't seem to be taking the match seriously."
He might have just seen it as a chance to experiment against a new type of player. Try some wild attacks and see it responds.
Of course, Lee Sedol might feel the same too.
@GJDrew, the only reason why I'm at 10% and not 1% is because I doubt Lee Sedol will be given any of AlphaGo's pre-match games. If my hunch is correct, this adds some element of uncertainty.
@GLDrew That's the only thing that worries me -that Lee Sedol won't take the match seriously. I don't know anything about Go, but people have said that Fan Hui was playing an unconventional strategy and not trying very hard to win. I watched the week-long competition between the University of Pittsburgh's computer "Claudico" and four elite online poker pros last summer. Heads-up No-Limit Hold'em. The poker pros won by an average of 12 bb, which was judged not statistically significant (though the win rates of a couple of them toward the end of the competition were considerably higher). The bottom line was that the pros could beat Claudico, but two of them seemed to be trying and two did not. Even the two who were trying could have done better by comparing notes. I learned that people don't always take these things seriously, even though a lot of people are watching and rooting for them. And I don't know anything about Lee Sedol.
I watched the same poker match! Some of the poker pros were definitely trying to figure out the weird game of computer (for their own later use) rather than trying to necessarily maximize win chances. Especially in poker, computers often discover new ways to play that are different than conventional wisdom and pros might see it as a good chance to explore.
@Jean-Pierre How confident are you in this AlphaGo prediction? Would you be willing to wager money in a bet? If so, at what odds?
I am trying to figure out how much I can learn from your very low ~10% prediction given your superforecaster status. You are clearly great at forecasting, but maybe in this particular case you didn't think as carefully about it as some of the other people estimating 60+% that AlphaGo will win (?), so I'm not sure how much I should believe your low estimate and lower mine from what it otherwise would be. Thanks.
I'm very confident in this forecast unlike, say, my Super Bowl forecast. I have thought about this one quite a lot. I just haven't posted my thoughts here (too time-consuming). Of the 50 or so questions (!) that I'm forecasting right now, I would rank this among my top 5 in terms of confidence-level. Would I wager money? If I were a betting man, I would.
My forecast is actually 5% now because Lee Sedol (9-dan pro) in a different league compared to Fan Hui (2-dan pro). Fan Hui would be very lucky to win 1 in 10 even games against Lee Sedol. And Fan Hui won 2 against AlphaGo. My sense is he under-performed and could have won more. Lee is also far better than the ordinary 9-dan pro and has a chance of becoming one of the greats. A weak 9-dan pro might be able to win 1 out of 5 against Lee. But, then again, maybe not. Although AlphaGo will improve before the match, I seriously doubt it will be playing at Lee's level any time soon. I would even give high odds (65%-75%) of Lee winning 5-0.
For expert commentary on AlphaGo's games see https://www.youtube.com/watch?v=NHRHUHW6HQE You might find it more helpful than my layman's analysis.
@WJK, don't change your forecast because of mine. My Brier scores have been awful (compared to the consensus) since the New Year. Frankly, I'm surprised I'm still on the GJP Classic Geopolitics Leaderboard.
@Jean-Pierre, how closely have you examined the question of the likelihood that AlphaGo will improve enough to reach Lee Sedol's level by March? I see this is the critical question. Yes, it's definitely quite a large gap to close, but AlphaGo is already a large step up from past Go programs, and the paper they published said they had several ways to improve. What makes you so sure that the improvements won't be large enough to close the gap?
Edit: Also what is your most recent forecast? Is it still 10%?
No, my recent forecast is 5%.
I'm skeptical of the idea that five or six months is enough time to close the gap. For one, it seems to me that AlphaGo still plays very much like a computer. For example, Myungwan Kim argues that AlphaGo has trouble timing moves and assessing the changing value of stones. These weaknesses are deadly. They are also typical of Go computers. And the persistence of these weaknesses suggests to me that Google, in spite of their accomplishments, still hasn't solved the main problem with Go-playing computers. Go is a game with lots of ambiguity. Much more than chess. Based on what I've seen so far, I'm convinced AlphaGo is a lot better when brute force is called for rather than intuitive reasoning.
Myungwan Kim also argues that AlphaGo really needs to play more games against 9-dan pros in order to improve to their level. If it keeps playing mostly against itself, its playing strength will plateau. I have no reason to think he's wrong on that point. His commentary is in the link I provided. I recommend watching it.
Thanks the response. I've watched the Myungwan Kim commentary video. I wonder though, if AlphaGo really was expected to still be significantly below Sedol's strength (such that his match-winning chances were only 5%) in March, does it really make sense that the DeepMind team would have issued the challenge date of the match for March upon publishing the paper in late January? Why not schedule it for April or May to give themselves more time to improve so that they actually have a chance in the match? It thus seems to me that the seemingly-early challenge date is evidence that they are actually stronger than you say.
Also, even if they wanted to challenge him soon in March just to get better feedback of their strength and publicize their success so far, would they really offer a $1 million prize if they thought their odds were only 5%?
So even if I were to agree with you that 5% is a good prior for what I would have assigned in October for the probability that AlphaGo would win a match in March against Sedol, it seems that the fact that upon publication of their paper in late January they announced the match just over a month away in March (as opposed to announcing it for April/May), that one should update this 5% prior significantly to make it higher. What are reasons for keeping it low (<25%)?
I don't want to read too much into the DeepMind team's actions.
My prior would be closer to 1%. My current range of 5%-10% takes into account improvements in AlphaGo's level of play since October. Of course, I don't know what its level of play will be come match time and I could be making a serious mistake. But Kim's arguments are sound. In the final analysis, it took 9 years for chess computers to go from beating an International Master for the first time to beating world champion Garry Kasparov. Maybe go computers will make the same leap in a much shorter timeframe. But five months is really ambitious.
Thanks for the comments and questions. I'll consider what you say.