Top 500 Data Redux

[Frau_Blucher]

Hey gang, apologies in advance for an over-the-top geekfest – Freewave’s awesome Top 500 effort yielded a data set that just screams out for further digging. To that end, I’ve run some correlation numbers on peoples’ selections and the people themselves to see what trends and relationships there might be. Thanks to Freewave for graciously sharing his data from the epic poll:

http://www.nwoutpost.com/forum/topic.asp?TOPIC_ID=6278

First, the raw numbers. I had to eliminate the 50-answer replies because they would have artificially skewed the tests, so apologies to those two folks. I also reassigned scores with 5 still going to 1-10 but only 3 for 11-30, 2 for 31-50 and 1 for 51-100. I just figured top 10 means you feel pretty strongly about it – it also gives more variety to the data set. Using this method, the top 100 for these inputs are somewhat different than the originals:

#—Sum—Song
1—39—Joy Division-Love Will Tear Us Apart
2—39—New Order-Blue Monday
3—35—Gary Numan-Cars
4—33—Talk Talk-Its My Life
5—30—Echo n The Bunnymen-The Killing Moon
6—26—B-Movie-Nowhere Girl
7—25—Visage-Fade to Grey
8—24—English Beat-Save It For Later
9—24—Psychedelic Furs-Love My Way
10—24—The-The-Uncertain Smile
11—22—Split Enz-I Got You
12—19—Church-Under the Milky Way
13—19—Icicle Works-Birds Fly Whisper To A Scream
14—19—Roxy Music-More Than This
15—19—Smiths-How Soon Is Now
16—19—Soft Cell-Tainted Love
17—18—Heaven 17-Let Me Go
18—17—Bongos-Numbers With Wings
19—17—Freur-Doot Doot
20—17—Yaz-Situation
21—16—ABC-The Look of Love
22—16—Bronski Beat-Small Town Boy
23—16—Cure-Inbetween Days
24—16—Real Life-Send Me an Angel
25—16—Records-Starry Eyes
26—15—Madness-One Step Beyond
27—15—Payola$-Eyes Of A Stranger
28—15—Ultravox-Vienna
29—14—A Flock of Seagulls-Wishing If I Had
30—14—Clash-Train In Vain
31—14—Cure-Boys Dont Cry
32—14—Echo n The Bunnymen-The Cutter
33—14—Industry-State of the Nation
34—14—Thomas Dolby-One of Our Submarines
35—13—Gary Numan-Are Friends Electric
36—13—Ministry-Revenge
37—13—Siouxsie n Banshees-Cities in Dust
38—13—U2-New Years Day
39—12—Buzzcocks-Ever Fallen in Love
40—12—Chameleons-Up The Down Escalator
41—12—Cure-Just Like Heaven
42—12—Devo-Gates of Steel
43—12—Eurythmics-Love is a Stranger
44—12—Human League-Dont You Want Me
45—12—Japan-Quiet Life
46—12—Modern English-I Melt With You
47—12—New Order-Bizarre Love Triangle
48—12—Pete Shelley-Homosapian
49—12—Re-Flex-The Politics of Dancing
50—12—Secession-Touch
51—12—Simple Minds-Someone Somewhere In Summertime
52—12—Smiths-There Is a Light That Never Goes
53—12—Tears For Fears-Change
54—12—XTC-Making Plans For Nigel
55—11—Alphaville-Forever Young
56—11—Aztec Camera-Oblivious
57—11—Berlin-The Metro
58—11—Big Country-In a Big Country
59—11—Blancmange-Dont Tell Me
60—11—Bolshoi-A Way
61—11—Cult-She Sells Sanctuary
62—11—Cure-Love Cats
63—11—Damned-Alone Again Or
64—11—Depeche Mode-Just Cant Get Enough
65—11—Duran Duran-Girls On Film
66—11—INXS-Dont Change
67—11—Specials-Ghost Town
68—10—A Flock of Seagulls-I Ran
69—10—A Flock Of Seagulls-Space Age Love Song
70—10—A Flock of Seagulls-Telecommunication
71—10—Buggles-Video Killed the Radio Star
72—10—Clash-London Calling
73—10—Culture Club-Do You Really Wanna Hurt Me
74—10—Cure-Lets Go to Bed
75—10—David Bowie-Ashes to Ashes
76—10—Eurythmics-Here Comes the Rain Again
77—10—Lords of New Church-Dance with Me
78—10—Ministry-Every Day is Halloween
79—10—New Order-Ceremony
80—10—New Order-Perfect Kiss
81—10—Pet Shop Boys-Love Comes Quickly
82—10—Pretenders-Mystery Achievement
83—10—REM-Carnival of Sorts Boxcars
84—10—Romeo Void-Never Say Never
85—10—Style Council-My Ever Changing Moods
86—10—Talk Talk-Talk Talk
87—10—Tin Tin-Kiss Me
88—10—When in Rome-The Promise
89—10—Yaz-Nobodys Diary
90—9—Cure-A Forest
91—9—Depeche Mode-Get the Balance Right
92—9—Echo n The Bunnymen-Silver
93—9—Gang Of Four-Damaged Goods
94—9—Jam-Thats Entertainment
95—9—Killing Joke-Love Like Blood
96—9—Lords Of New Church-Open Your Eyes
97—9—Pet Shop Boys-West End Girls
98—9—Peter Godwin-Images Of Heaven
99—9—Smiths-Big Mouth Strikes Again
100—9—Suburbs-Love Is The Law
101—9—Tears For Fears-Pale Shelter
102—9—Waterboys-The Whole Of The Moon

Now on to the juicy stuff…

[Frau_Blucher]

Having established the data set, the first thing I wanted to see was how different peoples’ responses might be similar. To do this accurately however, I needed every single response rather than just the top 100 or 500. This meant crunching all ~2600 individual replies and ~1600 unique song entries into a by-person pivot table. I took those results and ran them through a k-means clustering algorithm and asked it to place people into 5 bins or buckets. K-means is a fairly simple process that iteratively compares every single value in a data set thousands of times and sees how sets of them start to fall in with eachother. It doesn’t generate lots of correlation statistics but tends to yield very accurate qualitative data views.

Below are the five buckets showing the first 69 songs. The reddest boxes indicate songs that people scored highly and black spaces where they weren’t chosen voted for.

Ebbhead, Freewave, KYYX4ever, Dave, MikeP, Marie3, Shortysgrowingabeard, and Otters appear to have similar tastes by this measure. Egg-Roger, Oslo, ru480s2, and Yipyipcoyote are in another bin, etc. I’ll proudly wear the colors of Hexicon and Wozgo’s bin any day. The last bin to the right seems the most different, and we’ll get further evidence of that shortly.

<img src="http://www.pbase.com/myirwin/image/44717338.jpg">

But there are some limitations of this view…

[Frau_Blucher]

K-means is fine for bucketing things but you have to artificially choose how many buckets yourself, plus I wanted more numbers! Here’s another view, this time using Hierarchical Clustering with a Spearman Correlation Similarity Metric and Complete Linkage Clustering to measure the statistical distance between every single point in the data set. The Spearman Rank method is ideal for Ordinal Ranking situations (such as song scoring!) as opposed to naturally or randomly occurring data points. So here’s what we got – a clustered dendogram tree to indicate how closely correlated or apart the individuals are…

<img src="http://www.pbase.com/myirwin/image/44717340.jpg">

People here fall into roughly 4 clusters. In the first one, Ebbhead and Freewave show the highest coefficient of correlation at 0.298 – a normally pretty low number but quite high here given the small sample population and the huge number of possible song selections.
Manekineko fits into this subtree as well having a 0.211 Cc in common with the other two guys.

In the second subcluster over, Blir and Wozgo share a 0.182 Cc with MikeP close behind at 0.146 – damned good company, thank you very much! This subcluster falls in with Ebbhead’s subcluster at 0.080 Cc.

A third sub-cluster has Hexicon and Otters sharing a 0.212 Cc and dtRadio appearing in the same tree at 0.098 – different but still related. All three subclusters in Cluster 1 share a minimum Cc of 0.030 with eachother.

In Cluster 2, Dave and WolverineSyr are 0.179 correlated with Exrat coming next at 0.092 and Egg-Roger farther out at 0.059 but still within the same subcluster.

The next subcluster over hasBrock and KYYX4ever at 0.162 correlated, Marie3 at 0.091 and DayGloJo quite far out at 0.041. These subclusters have a 0.023 correlation to Cluster 2 as a whole.

In Cluster 3, Kimiii and Shortysgrowingabeard look like a 0.117 love connection with Rubbeet’s three’s-a-crowd score of 0.043 still keeping her in the same subcluster. XXX and ru480s2 are at 0.087 and Randy at 0.057 in a second subcluster that is only 0.011 related to the other subcluster.

Now come the outlyers. There is a fourth cluster that’s barely a cluster at all. Oslo and Yipyipcoyote have a coefficient of correlation of only 0.052. Skeggia meanwhile wins our individuality award with an astounding Cc of only 0.005 even to these guys! This cluster has essentially 0.0 correlation to any of the others. You freakin rebels!

[Frau_Blucher]

Now on to some songs that fell into clusters too, meaning they tended to be picked with each other by different people. This only works for popular songs since the rare ones would tend to cluster with the individual who picked other rare songs that no one else did either. There are four interesting groups of songs that are highly correlated and one song that has a surprisingly low correlation.

The first set consists of Fade to Grey, Cars, Tainted Love, and Don’t You Want Me. I can see how they’d be picked together…

<img src="http://www.pbase.com/myirwin/image/44717342.jpg">

Next are Nowhere Girl, Doot Doot, Milky Way, and Don’t Dream It’s Over. I can totally see it again – hey, the statistics don’t lie!

<img src="http://www.pbase.com/myirwin/image/44717344.jpg">

It’s My Life, Killing Moon, Blue Monday, Love My Way, and The Metro is another set that people are likely to have picked together in common…

<img src="http://www.pbase.com/myirwin/image/44717375.jpg">

Lastly, Starry Eyes, I Confess, I Got You, The Look of Love, and Ever Changing Moods don’t surprise either, but they do seem to have a different flavor than the other groups – maybe there’s actually something to this number crunching…

<img src="http://www.pbase.com/myirwin/image/44717390.jpg">

Now we get to an interesting point. These songs all had very high correlation coefficients within their respective groups as follows:

Songs1 - 0.713
Songs2 - 0.678
Songs3 - 0.713
Songs4 - 0.685

But the big cheese, Joy Division’s LWTUA didn’t fall strongly with any group in particular, with a 0.587 Cc to its nearest neighbors. What’s the deal? I’m thinking it’s a song that’s widely held in pretty high regard and that people will choose LWTUA without necessarily having lots of other things in common…except of course a common respect for LWTUA. It came in #1 and this statistical inference very likely supports its position as an overarching favorite!

[Frau_Blucher]

Okay, lastly, there may be other hidden statistics in this data. One thing I kept wondering about was how many times a particular song was chosen, not just its overall score. I mean, what if some songs got picked a lot but only as 1-pointers while some others got picked just a couple of times by coincidentally big fans. Here’s two more views; first with the Count of each song (total times picked) then after that, an interesting count-factored ranking…

#—Song—Count
1—Joy Division-Love Will Tear Us Apart—14
6—Visage-Fade to Grey—12
4—Talk Talk-It’s My Life—11
7—B-Movie-Nowhere Girl—11
2—New Order-Blue Monday—10
3—Gary Numan-Cars—10
8—The-The-Uncertain Smile—10
12—Soft Cell-Tainted Love—10
5—Echo & The Bunnymen-The Killing Moon—9
11—Split Enz-I Got You—9
39—Modern English-I Melt With You—9
13—Icicle Works-Birds Fly (Whisper To A Scream)—8
17—Heaven 17-Let Me Go—8
9—English Beat-Save It For Later—7
10—Psychedelic Furs-Love My Way—7
14—Roxy Music-More Than This—7
18—Freur-Doot Doot—7
21—Real Life-Send Me an Angel—7
26—Ultravox-Vienna—7
29—A Flock of Seagulls-Wishing (If I Had A Photograph Of You)—7
30—Thomas Dolby-One of Our Submarines—7
35—Gary Numan-Are Friends Electric?—7
36—U2-New Year’s Day—7
40—Eurythmics-Love is a Stranger—7
41—Re-Flex-The Politics of Dancing—7
55—Berlin-The Metro—7
15—Church-Under the Milky Way—6
19—Bongos-Numbers With Wings—6
22—ABC-The Look of Love—6
23—Bronski Beat-Small Town Boy—6
24—Cure-Inbetween Days—6
25—Records-Starry Eyes—6
27—Madness-One Step Beyond—6
28—Payola$-Eyes Of A Stranger —6
31—Echo & The Bunnymen-The Cutter—6
42—Human League-Don’t You Want Me—6
43—Pete Shelley-Homosapian—6
44—Tears For Fears-Change—6
45—XTC-Making Plans For Nigel—6
56—Cult-She Sells Sanctuary—6
68—Buggles-Video Killed the Radio Star—6
69—Ministry-Every Day is Halloween—6
90—Pet Shop Boys-West End Girls—6
91—Tears For Fears-Pale Shelter—6
129—Fiction Factory-Feels Like Heaven—6
16—Smiths-How Soon Is Now?—5
20—Yaz-Situation—5
32—Clash-Train In Vain—5
33—Cure-Boys Don’t Cry—5
34—Industry-State of the Nation—5
37—Siouxsie & Banshees-Cities in Dust—5
46—Buzzcocks-Ever Fallen in Love—5
57—Alphaville-Forever Young—5
58—Aztec Camera-Oblivious—5
59—Big Country-In a Big Country—5
60—Blancmange-Don’t Tell Me—5
61—Damned-Alone Again Or—5
62—Duran Duran-Girls On Film—5
70—Culture Club-Do You Really Wanna Hurt Me?—5
71—Lords of New Church-Dance with Me—5
72—Romeo Void-Never Say Never—5
92—Killing Joke-Love Like Blood—5
93—Peter Godwin-Images Of Heaven—5
103—Furniture-Brilliant Mind—5
104—Tears for Fears-Mad World—5
105—Wire Train-Chamber Of Hellos—5
130—Crowded House-Don't Dream It's Over—5
131—Jo Boxers-Just Got Lucky—5
132—Shriekback-All Lined Up—5
169—Polecats-Make a Circuit with Me—5
170—REM-Radio Free Europe—5

There’s some interesting movers, especially Melt With You rocketing up with a large number of counts but apparently few Points. So how about we Factor the Counts into the Points? We don’t want to skew the results completely – maybe use the count as a small multiplier on the points? So here’s the points multiplied by 10% of the count value. I think there’s some impressive differences that MAY potentially reflect the group sentiment more accurately…or not, maybe just differently. It could have made an interesting tie-breaker in any case…

#—Song—Factor
1—Joy Division-Love Will Tear Us Apart—54.6
2—New Order-Blue Monday—39
4—Talk Talk-It’s My Life—36.3
3—Gary Numan-Cars—35
6—Visage-Fade to Grey—34.8
7—B-Movie-Nowhere Girl—28.6
5—Echo & The Bunnymen-The Killing Moon—27
8—The-The-Uncertain Smile—24
11—Split Enz-I Got You—19.8
12—Soft Cell-Tainted Love—19
9—English Beat-Save It For Later—16.8
10—Psychedelic Furs-Love My Way—16.8
13—Icicle Works-Birds Fly (Whisper To A Scream)—15.2
17—Heaven 17-Let Me Go—14.4
14—Roxy Music-More Than This—13.3
18—Freur-Doot Doot—11.9
15—Church-Under the Milky Way—11.4
21—Real Life-Send Me an Angel—11.2
39—Modern English-I Melt With You—10.8
26—Ultravox-Vienna—10.5
19—Bongos-Numbers With Wings—10.2
29—A Flock of Seagulls-Wishing (If I Had A Photograph Of You)—9.8
30—Thomas Dolby-One of Our Submarines—9.8
22—ABC-The Look of Love—9.6
23—Bronski Beat-Small Town Boy—9.6
24—Cure-Inbetween Days—9.6
25—Records-Starry Eyes—9.6
16—Smiths-How Soon Is Now?—9.5
35—Gary Numan-Are Friends Electric?—9.1
36—U2-New Year’s Day—9.1
27—Madness-One Step Beyond—9
28—Payola$-Eyes Of A Stranger —9
20—Yaz-Situation—8.5
40—Eurythmics-Love is a Stranger—8.4
41—Re-Flex-The Politics of Dancing—8.4
31—Echo & The Bunnymen-The Cutter—8.4
55—Berlin-The Metro—7.7
42—Human League-Don’t You Want Me—7.2
43—Pete Shelley-Homosapian—7.2
44—Tears For Fears-Change—7.2
45—XTC-Making Plans For Nigel—7.2
32—Clash-Train In Vain—7
33—Cure-Boys Don’t Cry—7
34—Industry-State of the Nation—7
56—Cult-She Sells Sanctuary—6.6
37—Siouxsie & Banshees-Cities in Dust—6.5
68—Buggles-Video Killed the Radio Star—6
69—Ministry-Every Day is Halloween—6
46—Buzzcocks-Ever Fallen in Love—6
57—Alphaville-Forever Young—5.5
58—Aztec Camera-Oblivious—5.5
59—Big Country-In a Big Country—5.5
60—Blancmange-Don’t Tell Me—5.5
61—Damned-Alone Again Or—5.5
62—Duran Duran-Girls On Film—5.5
90—Pet Shop Boys-West End Girls—5.4
91—Tears For Fears-Pale Shelter—5.4
38—Ministry-Revenge—5.2
70—Culture Club-Do You Really Wanna Hurt Me?—5
71—Lords of New Church-Dance with Me—5
72—Romeo Void-Never Say Never—5

So what’s the verdict? Too much? Not enough?? [:D]

[medication881]

Dude, you need to come onto my radio show...

/not kidding

[8D]

[SwampThing]

Looking forward to analysis on the Top 80 albums!

[Arcadia]

Did anyone else pop a boner (it's different for women I know) while reading Blir's 5 posts?

My only question is.......Bongos - Numbers With Wings.......WTF?

[eight6]

Blir:

Any chance you could do my taxes?

<img src="http://img.photobucket.com/albums/v647/dbince/eight.gif"><img src="http://img.photobucket.com/albums/v647/dbince/spinning6.gif">

[ru480s2]

[quote][i]Originally posted by Arcadia[/i]
<br>Did anyone else pop a boner (it's different for women I know)
[/quote]
I nearly pee'ed my pants. And from what I've heard, I'm sure some women got wet too.

[2Nu]

Still glassy eyed and in awe of that report, whatever it meant.
I have no F**king clue how you came up with all that but SERIOUSLY, I think you missed your calling to work for <a href="http://www.nielsenmedia.com/" target="blank">Nielson</a> or <a href="http://www.arbitron.com/onlineradio/home.htm" target="blank">Arbitron</a> At least be a consultant!

[WolverineSyr]

I forwarded this off to my co-worker who sits right next to me. She has a Masters in Math, and loves new wave.

She's in heaven, absolute heaven.

She's spent some time explaining this to me - hey, what's work for anyways?

[Frau_Blucher]

[quote][i]Originally posted by WolverineSyr[/i]
<br>I forwarded this off to my co-worker who sits right next to me. She has a Masters in Math, and loves new wave.

She's in heaven, absolute heaven.

She's spent some time explaining this to me - hey, what's work for anyways?
[/quote]

LOL! But oh crap, I don't know if this stuff can actually pass muster with an honest-to-god mathematician! I'm just a biologist by training. But lots of my biotech customers have to do this stuff more and more for things like micro-array gene expression experiments where they study 30,000+ genes across 100,000+ RNA transcripts using a million DNA probes to cover the 3 billion base pairs of the human genome...and that's just one sample. Multiply that by a fairly small two-dozen sample experiment done in triplicate and you end up with 72 million data points...now THAT's crazy!

Frayo, are you listening? There's big bucks for you math guys in biology!

[Frau_Blucher]

[quote][i]Originally posted by kimiiii[/i]
<br>I had no idea that our haphazard lists would be so scrutinized. I would've used a much different formula for mine. As it is, I used the following quasi-scientific theorem as my ranking guideline:

"I like it. It's got a good beat and you can dance to it."
[/quote]
Actually Kimiii, that's what I do too! In fact, most analyses prefer random and unbiased inputs without [i]a priori[/i] knowledge of outcomes or analysis methods, whose results should only be judged on an epi-phenomenal basis. [:D]

Oh, and BTW, it's definatley NOT brainpower...just the amazing, everyday personal computer!

[NuWaveRx]

Blir,
Oustanding population analysis, although I must confess, even with training in biostats you lost me a couple of times. Now if you had some demographic data (e.g age, sex, where they live, SE status, where they grew up) you could do some regression analysis to find some interesting things. (e.g males, age 35-40 who grew up in SoCal are more likely to put X band as their top choice). etc. Very well done!