The Problem With Populism, Part Two

In a previous post, I discussed the IMDB Top 250 List, and why it's essentially meaningless and not a useful tool for comparing movies to each other. In short, the problem with it is populism, everyone gets a say, no matter how wrong they are about it. Additionally, films that predate the internet face a massive handicap in terms of exposure and the number of rankings they achieve. Populism is a complex, multi-faceted problem, but one way to simplify things is to simply adopt clearly defined terms for the adjudication of films, terms that put everyone on the same page. This would eliminate 100 people with 100 different favorite films from shouting at each other.

Now, perhaps I've been unfair and the IMDB list deserves to be defended. It is based on a weighted ranking system that accounts for varying numbers of votes,after all. As IMDB discloses...

The formula for calculating the Top Rated 250 Titles gives a true Bayesian estimate:

weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C
Where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 250 (currently 25000)
C = the mean vote across the whole report (currently 7.0)
For the Top 250, only votes from regular voters are considered.

So let's compare the math between The Shawshank Redemption (#1, 9.3/10) and Citizen Kane (#63, 8.5/10).

So, this isn't a math blog, but I did do all the calculations, per IMDB's disclosure, and my result was not 9.3, but rather 9.06 (-0.24).

Likewise, for Citizen Kane (the most frequently cited "best film ever made"), the result, factoring for 233,777 votes - less than 1/5 of the votes of Shawshank - was not 8.5 but rather 8.08 (-0.42). So both movies dropped, but the movie with significantly fewer votes dropped more. I'm intrigued. Let's look at another set of movies with a similar disparity.

The Dark Knight is #4 on the list, with a 9.0 rating and the second highest number of votes. Do the math and the 9.0 rating falls to 8.86 (-0.14). That's not much of a drop.

The Third Man though, #98 with an 8.4 rating has only managed to get 88,061 votes. The math shows a weighted average of 8.01, a drop of 0.39 points.

I think I'm seeing a trend here. But why? Is it a conspiracy? Not really. The idea is just that the fewer votes a film has, the closer it gets pushed toward the report mean, in this case, 7.0. The more votes a film gets, the closer the weighted average gets pushed toward the actual mean rating for that particular movie. For The Dark Knight the mean rating is 8.9.

What does it all mean? Well, it comes back around to the fact that populism is inherently unfair and is no way to decide the merits of art or commercial products, let alone things that are in fact a combination of both of those things. Just for fun, I scaled The Third Man to see what would happen if it had the same number of votes as Shawshank (while maintaining the same proportional distribution) and the equation works out to 8.23 instead of 8.01. That's a hefty difference when the only thing that changed is the volume of votes.

This is all rather pointless though. If you've been paying attention you'll have noted that none of the results I've gotten have matched the stated average ratings. There's a problem somewhere, and IMDB is aware of it. They put it there.

IMDb publishes weighted vote averages rather than raw data averages. Various filters are applied to the raw data in order to eliminate and reduce attempts at 'vote stuffing' by individuals more interested in changing the current rating of a movie than giving their true opinion of it. The exact methods we use will not be disclosed. This should ensure that the policy remains effective. The result is a more accurate vote average.

So they basically just hide the way they actually figure the ratings out. Lest you think I'm just making this up, CLICK HERE.

This is all so very Don Quixote of me, I'm well aware. My quixotic screaming into the internet demanding justice for old movies is just a modern way of tipping at windmills. I won't get IMDB to change anything. But, maybe, just maybe, I can get more people to watch and then rate these old movies and restore their glory that way. Or, on a much less ambitious note, perhaps I can just get a few kids to realize the difference between popularity and merit, and understand that the IMDB Top 250 List is deeply flawed and shouldn't be taken too seriously. I'm willing to settle for that.