Evaluating Jones' SoccerRatings and the RPI

SciFi, I am not sure investigations would produce much more than we already know about bias in women's soccer. Now, if money were changing hands that would be a horse of a different color.

Also, as you state, anything to do with basketball or the fairly recent BCS, you would have coaches and sports commentators clammoring across the nation.

UPSoccerFanatic wrote:SciFi, one thing I should emphasize, which I do when I'm discussing this with people not from the West Region, is that the RPI's discrimination is not necessarily a West Coast/East Coast thing. It is a strong region/weak region thing. In essence, the RPI always will discriminate against strong regions and in favor of weak regions. If I were going to really characterize the RPI, I would say that in rating teams for participation in the NCAA tournaments and getting seeds, it favors distributing them on a geographically balanced basis in proportion to the number of teams in each region. This is directly inconsistent with an explicit NCAA policy that says the best teams should participate and get seeds, regardless of conference (substitute "region"), which I take to mean that it's irrelevant if a bunch of at large teams (or seeds) come from one conference or, conversely, no at large teams (or seeds) come from a conference. The reason the West Region gets the shaft these days in Women's Soccer is that the West Region, by a lot, is the strongest region. If that ever changes, then it will be some other region getting the shaft.

The only way there would be blanket discrimination would be if the West Region were the strongest in all sports. I'm sure that isn't true right now in lacrosse and ice hockey.

This is why I put "East Coast Bias" in quotes. There just is no solid proof of this. However, many fans do carp about "East Coast Bias" in basketball and in football, which is why I mention it at all.

That there is bias in college sports, is a reasonable conclusion to make. You've proven it in Women's Soccer. I'm just making the suggestion that the best way to put pressure on the NCAA to address said bias, is to involve the press. The only way to involve the press seems to me to be to prove bias in Men's Basketball and in Men's Football (BCS). The press (and the NCAA) simply won't care about bias in Women's Soccer; but, by showing bias accross the board with all sports (or with a significant number of sports), you can pressure the NCAA to change THE WHOLE system. Thereby getting the system change in Women's Soccer. I, again, applaud your efforts and encourage you to continue and to broaden your investigation. I really wish I could help, but I am just not at that level when it comes to math.

Auto Pilot wrote:SciFi, I am not sure investigations would produce much more than we already know about bias in women's soccer. Now, if money were changing hands that would be a horse of a different color.

Also, as you state, anything to do with basketball or the fairly recent BCS, you would have coaches and sports commentators clammoring across the nation.

Others may disagree with me, but I have, for a long time, suspected that there are and have been shenanigans going on in the NCAA. I think investigations could turn up A LOT of shady activity.

-SciFi

Today, I worked on analyzing the Massey rating system. Massey's system is important because it is one of the systems included in the "computer" rating used for football BCS purposes. The result is extremely interesting, at least to me. Massey's and Jones' ratings' correlation with actual 2008 regular season game results is exactly the same. In fact, although their team rankings do not match exactly, in every game (of the more than 3,000 total games) they had the same team with a higher ranking.

I also should note that although I reported Jones as having a 73.3% correlation with actual game results, that is based on comparing his ratings covering games through the first weekend of the NCAA Tournament to results through the end of the regular season. Since the ratings were based on a slightly different set of games than the games I applied them to, I would expect that Jones' correlation would be slightly better if I'd been able to apply ratings through the end of the regular season to the regular season's games. (Jones did not publish ratings through the end of the regular season.) I had the same problem with Massey.

So, Jones, Massey, and the unadjusted (and adjusted) RPI have equally good correlations to regular season results as a whole. However, although Jones and Massey both are like the RPI in that they overrate some regions and underrate others, their overrating and underrating are significantly less than those of the RPI.

Add my thanks for the great work UPSF.

One question with respect to regional bias: What if regions vary in the quality of their teams that undertake interregional play? For example, suppose hypothetically that in the West region only the best teams play Eastern teams and the weaker teams do not. For example, it's probable that Portland, UCLA, Stanford, etc. play a more "national" schedule than Gonzaga, Oregon, St. Mary's, etc. But further suppose that this is less true in other regions, perhaps because the distances are smaller. In the East, many schools can get to an inter-regional game on a bus.

This would seem to lead to the West teams having a better inter-regional record than teams in the other region simply because only the best teams in the West are playing inter-regionally.

So I have two questions that I could probably answer with enough head scratching over your formulas but that you can surely answer directly:

1. Do these differences exist?
2. Do they matter for your regional RPI rankings or for the Jones and Massey ratings?

Forget what I said about Massey's ratings last night, I made an error and it turns out his rating strength comparisons do not match exactly with Jones. In addition, Massey's ratings correlate very slightly better with actual results than Jones'. That means I have to do separate computations for how Massey treats the regions.

Soreknees, your questions are interesting. You're absolutely correct that for the Mid Atlantic region, as an example, it abuts three other regions which makes having inter-regional games relatively easy. And, as you point out, the travel distances are much smaller so that teams with low budgets (presumably some of which are lower-ranked teams) are more able to travel. However, as I think about it, why would it explain one region's teams out-performing the RPI and other regions' teams under-performing? At least as far as I've gotten, I don't think it would.

The reason I thought it might is that inter-regional comparisons by necessity rely on the outcomes of inter-regional games. If one region only sends its best teams to inter-regional games, then it will have a better record than regions that send a balanced mix of teams to play other regions. This wouldn't be a factor if only the top teams from the West played Eastern teams as long as they only played the top teams in the East. But if the top teams in the West were feasting on weaker Eastern teams and that was the only inter-regional play for the West Region, then it seems like it could give an overly optimistic picture of the West's quality.

Does this make sense or am I worried about nothing?

It seems that what UPSF is talking about as I see it is the difference between what the rating system say about who should win and who actually does.

If a strong West team plays a patsie, the rating system should reflect a win for the West team. If it plays a supposedly stronger team, it should lose.

Apparently UPSF has shown that many more "upsets" happen than should when the West plays out of region. That would indicate a skewed rating system.

And a rating system that takes regional diferences into account should show the same percentage of upsets whether a team is playing interregional games or intraregional games.

Thinking about this some more, it seems to me a comparison of accuracy should only include perhaps the top 100 teams or so.
(maybe 100 isn't the right number, but it would show more than ranking the whole field)

The stated purpose of the RPI is to select the best teams for competition in the tournament, so it makes no difference if the RPI is correctly ranking teams that have no chance to make the tournament. Whether a team is ranked correctly at 320 or 267 is of no concern as far as the Championship is concerned.

That would presumably be the top 50 or so which currently get in because of their ranking, and you could double that to be sure you are counting all possible candidates.


I suppose that might exclude some automatic qualifiers, but they'd get in no matter what their RPI was.

SoreKnees, the Geez is right. The RPI is supposed to take into consideration both winning record and strength of schedule and to balance them correctly so as to rank teams properly based on both of them. So, if top West Region teams are feasting on weak Mid Atlantic Region teams, the West Region teams' excellent winning records will be balanced out by their poor strengths of schedule. That is the fundamental theory of the RPI and it is what the NCAA claims the RPI does in as fair and accurate a way as is reasonably possible. So, as Geez says, if the RPI is working correctly, when you look at how West Region teams are doing in relation to their RPI rankings and compare that to how other regions' teams are doing, you would expect the West Region's teams to be winning the same percentage of games on average as the other regions' teams are winning regardless of who they have been playing in their inter-regional games since their ranking already is supposed to have taken into account who they have been playing.

Geez, I agree with you that the testing process should look at the top 100 teams, too, probably broken down into some discrete groups to allow an analysis based on likely seeded teams and likely at large teams. That's on my "to do" list to see how the various rating systems treat those teams on a regional basis. I've already done it overall (as distinguished from regionally) for the unadjusted, adjusted, and non-conference RPIs as well as for my first run at a non-regional RPI (but not for Jones or Massey). I can tell you that looking only at the top 100 as ranked by each system, the rankings' correlations with game results are slightly better than the correlations when looking at all 318 teams, but higher by only a couple of percentage points. Further, when looking at the correlations for groups of ten (teams ranked 1-10, 11-20, 21-30, etc.), the correlations are best for the higher ranked teams and poorest for the lower-ranked teams.