Rob (robyrt) wrote in dreamermaxi,

Rock Band: Biased?

Wikipedia's comprehensive list of Rock Band songs include the difficulty ratings that developer Harmonix has assigned to each song for each instrument, as well as its overall difficulty. When I read a review complaining that the vocals had been marginalized compared to the guitar, bass and drum parts, I decided to run the math and see whether the vocal difficulty mattered as much as, say, the guitar difficulty when considering the song as a whole. While there is some evidence, the more interesting conclusion was the accuracy of Harmonix's ratings for a song compared to their ratings for a particular instrument. Rhythm games commonly suffer from misleading difficulty ratings, so here's a rare glimpse into how often this phenomenon really occurs.

Each track is rated from 1 to 9 in difficulty, from "Warmup" (1) to "Impossible" (9). After running the numbers, a surprisingly reassuring picture emerged: songs were underrated about as often as they were overrated, and only 35% of songs were over- or underrated by a full point or more. Among the instruments, the overall rating corresponded very closely to the ratings for guitar and drums, with bass rated 0.38 easier and vocals 0.53 harder per song.

The outliers, more likely to stick in people's minds, were more telling. Vocals were the most likely to wildly disagree with the overall rating - for example, "Mississippi Queen" is rated a 1 overall but an 8 for vocals, a nasty surprise for karaoke fiends. The two largest outliers, "Sprode" and "Shake," were distributed as a magazine-only promo, so their clearly wrong ratings are unlikely to be seen by casual players. Some examples of bad ratings are legitimate - the Aerosmith cover "Train Kept A-Rollin" is rated a 5 overall but scores at least 7 in every individual instrument - but far more common are group ratings that differ from the average of their instruments in an attempt to hew more closely to the drum track.

In defense of Harmonix, this bias towards the drums and against the vocals reflects a deeper understanding of the game mechanics than the numbers suggest. The drum track, dense and repetitive, makes it easy for a difficult pattern to "kill" a player within just a few seconds. Conversely, even the most difficult vocal tracks are judged by long phrases, so a struggling singer can give his band plenty of opportunities to "save" him. The drummer, more sensitive to changes in difficulty, therefore has his personal difficulty level reflected more closely in the overall number.

(Food for thought: The differing ability of instruments to "save" ailing bandmates also plays well into the likely demographics of Rock Band owners: Guitar Hero veterans are expected to save their super meter to rescue the comparatively passive drummer and singer.)

The top ten offending songs, with promo songs excluded:
  1. Train Kept A-Rollin - Official score 5, average score 8
  2. Gimme Shelter - Official score 2, vocal score 8
  3. Detroit Rock City - Official score 7, average score 4.75
  4. Reptilia - Official score 7, average score 4.75
  5. Go with the Flow - Official score 6, average score 3.75, drum score 7
  6. Seven - Official score 2, vocal score 8
  7. When You Were Young - Official score 3, average score 5
  8. Mississippi Queen - Official score 1, vocal score 8
  9. Outside - Official score 7, bass score 1
  10. Pleasure - Official score 6, average score 4, guitar score 9
  • Post a new comment


    default userpic