tag:blogger.com,1999:blog-38600807.post4919615842144525964..comments2023-11-05T04:16:44.937-05:00Comments on Advanced Football Analytics (formerly Advanced NFL Stats): The Problem with Power RankingsUnknownnoreply@blogger.comBlogger21125tag:blogger.com,1999:blog-38600807.post-30127592509390130802011-10-20T17:11:19.271-04:002011-10-20T17:11:19.271-04:00' Tuesday, October 18, 2011
Tom said...
...' Tuesday, October 18, 2011 <br />Tom said...<br /><br /> I agree entirely with your sentiment, Brian, but I have to admit that the model, relative to others, is not doing well this year. This season has been very predictable so far, but the model here has struggled to capture that predictability.'<br /><br />http://www.theunticket.com/wp-content/uploads/media/101911/camera2.gifRaHhttps://www.blogger.com/profile/14636068536257498024noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-67269885968235848582011-10-19T22:01:00.783-04:002011-10-19T22:01:00.783-04:00Agreed about the sample size. I was just curious s...Agreed about the sample size. I was just curious so I compiled all the data I could find. I'll probably continue to track this. Should I bother to post the results here when the sample size is larger?AESnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-47025440713041852492011-10-19T13:02:52.826-04:002011-10-19T13:02:52.826-04:00Being 2-6 (or worse) over 8 games would happen 14%...Being 2-6 (or worse) over 8 games would happen 14% of the time if we were just flipping coins. The sample size is too small to say anything about whether Brian's method is better/worse than the consensus.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-81228229266976755562011-10-18T23:09:05.767-04:002011-10-18T23:09:05.767-04:00As a very primitive test, there have been 8 games ...As a very primitive test, there have been 8 games so far where Brian's model has predicted an outcome that is different from the consensus prediction of the computer models tracked at predictiontracker.com. Brian's model is 2-6 in those 8 games.<br /><br />Week 4: Min at KC (W), NYG at Ari (L), Atl at Sea (L), NE at Oak (L)<br />Week 5: Cin at Jac (L), SD at Den (L)<br />Week 6: Car at Atl (L), Buf at NYG (W)AESnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-90856613931359122252011-10-18T17:07:34.343-04:002011-10-18T17:07:34.343-04:00Obviously the standings essentially capture the sa...Obviously the standings essentially capture the same thing I'm describing, with slight mental adjustments for strength of schedule and reputation, which is why they are pretty dumb. But I'd think of the power rankings more as a proxy for RPI or BCS style standings than for predictive accuracy.Davidnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-19327900649565627312011-10-18T17:01:16.422-04:002011-10-18T17:01:16.422-04:00Power Rankings aren't meant to be predictive. ...Power Rankings aren't meant to be predictive. They are meant to indicate who is having the best season at a particular moment.<br /><br />The outcomes of plays in the last couple minutes of a tied game don't have much predictive power, but they can have a massive effect on who is having a good season (Niners) and who is having a lousy season (Cowboys), which is what the power rankings capture.Davidnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-17387826272952830902011-10-18T16:31:37.249-04:002011-10-18T16:31:37.249-04:00Great article.Great article.Jacob Stevenshttp://www.absenceofevidence.comnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-85706051865905674642011-10-18T12:13:34.746-04:002011-10-18T12:13:34.746-04:00I tested the quality in terms of calibration, conf...I tested the quality in terms of calibration, confidence, and absolute error. I don't have the results to hand, as I'm not at my PC, but they were as suggested.<br />I also found that when Brian's model tended to disagree strongly with more traditional models, it was more often to his model's disadvantage that it did so. His model was also overconfident late in the season, though from what I understand he has rectified that somewhat now.Tomnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-38352481300256441882011-10-18T12:08:41.001-04:002011-10-18T12:08:41.001-04:00"I have looked back at the numbers, my own, a..."I have looked back at the numbers, my own, and those of others, and found that there is no significant difference in quality between Brian's numbers and those of other quality prognosticators."<br /><br />How exactly did you test the quality, and what were the actual results?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-41653505096383529642011-10-18T11:39:58.701-04:002011-10-18T11:39:58.701-04:00Maybe this article is better suited for the NY Tim...Maybe this article is better suited for the NY Times or Washington Post, but as someone commented last week, most readers of this site are smart enough to understand that the team with the best record doesn't always win. Or maybe traffic coming from those sites is dumbing down the readership here. It's not much of a stretch to believe that your rankings are more predictive than most subjective power rankings, but I don't know that you've provided much evidence that it's any more predictive than dozens of other objective statistical models.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-65905013221352097412011-10-18T11:21:24.256-04:002011-10-18T11:21:24.256-04:00Actually, I understand extremely well how these th...Actually, I understand extremely well how these things work. You are thinking 'oh, he thinks picking games is the key to a good model'. No, I don't, I know that the combination of model confidence, and model calibration, is what makes a model good, and I know, I have looked back at the numbers, my own, and those of others, and found that there is no significant difference in quality between Brian's numbers and those of other quality prognosticators.<br />As for three weeks of games being a small sample, it is. I agree, so we shall see how that picks up. However, those who have been predicting games from week one have been predicting at a higher average rate, including those early weeks, than Brian's model, and that should be food for thought, because there is clearly data there that is being thrown out, and I hate to see so much potential go to waste.<br />I have spent many a comment defending the model, but I think it is time for an overhaul, because it has more potential than is being realised.Tomnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-20904225764105903602011-10-18T11:03:02.897-04:002011-10-18T11:03:02.897-04:00"This season has been very predictable so far..."This season has been very predictable so far, but the model here has struggled to capture that predictability."<br /><br />You just don't understand how things work, do you?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-22659599141005089682011-10-18T10:52:46.637-04:002011-10-18T10:52:46.637-04:00You've got to be kidding. Based on 3 weeks of ...You've got to be kidding. Based on 3 weeks of games?Tednoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-33880689830165471572011-10-18T10:42:39.678-04:002011-10-18T10:42:39.678-04:00In comparison to the system average at thepredicti...In comparison to the system average at thepredictiontracker it is not doing all that well. It may be within three or four, but that is a large difference over a few weeks. I deliberately picked the system average as it doesn't suffer from the luck bias that individual systems can - there are those that have done better.<br />My point was more that, whilst it is easy to claim that these statistics are intrinsically better predictors, they do not perform any better than those using the more common statistics in intelligent ways. Whether this simply means that you can capture as much with points and yardage as you can with ratestats, or that the current use of these ratestats is inadequate I cannot say, but I can say that at present this model, whilst interesting, and potentially powerful, is not separating itself from the crowd.Tomnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-13202860862851172062011-10-18T10:33:08.312-04:002011-10-18T10:33:08.312-04:00That's simply not true. Brian's model is w...That's simply not true. Brian's model is within a couple games of Vegas favorites.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-89326801979062093652011-10-18T10:25:42.677-04:002011-10-18T10:25:42.677-04:00I agree entirely with your sentiment, Brian, but I...I agree entirely with your sentiment, Brian, but I have to admit that the model, relative to others, is not doing well this year. This season has been very predictable so far, but the model here has struggled to capture that predictability.Tomnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-71847638183506701362011-10-18T09:53:05.966-04:002011-10-18T09:53:05.966-04:00I sense this is a preemptive justification for Dal...I sense this is a preemptive justification for Dallas being atop the week 7 power rankings! And, I would agree.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-19360851213492885032011-10-17T23:59:55.640-04:002011-10-17T23:59:55.640-04:00Chase Stuart's point about QB picks being very...Chase Stuart's point about QB picks being very random year-to-year back through all 16-game seasons remains true this year -- Brady had only 4 all last year, already has 8 this year including 4 in one game -- and was true back through the 14- and 12-game-season eras as well.<br /><br />Back when there were a lot more interceptions thrown than today, Bart Starr won his one MVP award in 1966 on the back of the miracle achievement of throwing only three picks for the year -- the all-time low number.<br /><br />The next season he started by throwing 9 in his first two games, and for the full year threw more picks than any other QB in the league. Same QB, same teammates on offense, same system, same coach (same end result winning the Super Bowl). Go figure. There's lot of randomness in this game.Jim Glassnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-26287258668909709802011-10-17T21:22:46.139-04:002011-10-17T21:22:46.139-04:00There's definitely something to that. I've...There's definitely something to that. I've <a href="http://www.advancednflstats.com/2010/11/how-random-are-interceptions.html" rel="nofollow">looked at year-to-year int correlation</a>. It's very low, but so is intra-season int rate correlations.<br /><br />Here's <a href="http://www.pro-football-reference.com/blog/?p=6068" rel="nofollow">more from Chase Stuart</a>.Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-65513499907724493722011-10-17T21:11:49.441-04:002011-10-17T21:11:49.441-04:00I'd be interested to know the answer to that q...I'd be interested to know the answer to that question as well.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-34748092997805956672011-10-17T20:51:40.299-04:002011-10-17T20:51:40.299-04:00Not that this is necessarily feasible or a can of ...Not that this is necessarily feasible or a can of worms you want to open, but would adding a QB's career INT rate (for veteran QBs) as an input to the model help its accuracy or not? It seems that because INTs are so rare (sample so small), your model has to regress YTD INT rates heavily to league average -- and correctly so, if you are limited to current-season inputs. But I would think that long-term INTs rates are fairly predictive on a player level, and so would possibly account for some of the observed prediction error.<br /><br />The other way to ask this is: do you think your model tends to underrate teams with long-term low-interception QBs, and overrate teams with long-term high-interception QBs, because the model is not allowed to know info. from past seasons and thus has to be very conservative in projecting abnormal INT rates?Anonymousnoreply@blogger.com