Motley Crue and Javascript D3

A few months ago, I posted a line graph showing the drastic increase in popularity of Queen and their song “Bohemian Rhapsody on Spotify. Well, Motley Crue has a movie out, too.

crueCurrentPopEtc.png
Figure 01: The Dirt was released on Netflix March 22

That’s just their Spotify Popularity score which, under most circumstances, is pretty useless. When the popularity score makes a significant change outside of the normal Spotify popularity ebb and flow (more on that later), it can make graphs that rock like the one above.

Before we get to their Spotify Followers and some techie stuff, here’s how the movie affected their albums’ popularity.

albumsMonth.png
Figure 02: Between March 17 and April 17, their albums also jumped up quite a bit.

The above was made with a little help from Photoshop. I don’t yet know how to make split columns like that with D3.js but the data is just as real.

crueFormattedFollowers.png
Figure 03: I started tracking Motley Crue‘s followers March 17, five days before Dirt‘s release.

Creating the above graph for Spotify Followers was more of a pain in the butt than I thought it would be. I thought all I’d have to do was replace my pop variable with followers. They’re even in the same table now, so … you know, right? I wanted to have multiple artists represented with multiple lines showing just how drastic that steep line is. But the range of popularity and followers for the artists I track in my database is HUGE so, even after narrowing down the artists, the Y-Axis went from  1 million to 30 million and the lines were all flat — or, at least, that’s how they looked unless I were to make the graph ten feet tall like this quickly made column graph showing Followers for this year’s Rock and Roll Hall of Fame nominees that’s ugly for all kinds of reasons. I totally thought I was doing something wrong.

friends.png
Figure 04: Lame, lame, lame.

Even just sorting the followers column and grabbing the artists above and below Motley Crue didn’t really help.

phpMyAdminPopFollow.png
Figure 05: I wanted to include artists in recent news as a point of reference but Queen and the Stones (let alone Eminem) blow the curve.

I have to say I fell in love with this little query just now:

SELECT p.pop, p.followers, a.artistName
FROM popArtists p
JOIN artists a ON a.artistID = p.artistID
WHERE date = '2019-04-17'
ORDER BY followers DESC

It was so much easier to write than the query I wrote while relatively inexperienced to get the most recent data. Anyway, my array of artists got so small and still looked so lame I gave up on that idea but I still had this problem to solve:

crueUnformattedFollowers.png
Figure 06: You’re looking to the left at the Y-Axis, that’s what’s different.

At first, I thought that would be easy. Just like I thought the whole thing would be. The “Motley Crue and Friends” multi-line chart was based on a more recent project. I learned how to make multi-line charts a while after I made the single-line version and, by that time, I’d also learned to stop doing this:

const w = 1100;
const h = 400;
const padding = 40;

and start doing this:

var w = 1000;
var h = 400;
padding = {
top: 50,
right: 30,
bottom: 350,
left: 100
};

The latter chunk of code makes everything delightful. Specifically, I could add some padding to just the left side, easily making room for those huge numbers in the tens of millions for “Motley Crue and Friends” because that file used a padding object with four properties instead of a single padding variable. When I increased the padding for what I guess I’ll call “Motley Crue All by themselves” it squished the graph from every direction making it even more hideous than it is up there if you can imagine that. Don’t. Really, don’t imagine that.

Originally, I had this humble bit o’ code:

const yAxis = d3.axisLeft()
.scale(yScale);

… which was fine for a simple 0-100 scale. To go from the horrifying Y-Axis in Figure 06 to the downright sexy Y-Axis in Figure 03, I learned to do this:

formatMillions = d3.format(".3s");
const yAxis = d3.axisLeft ()
.scale (yScale)
.tickFormat (function(d) { return formatMillions(d) } );

First, I looked through the epic-length documentation for D3-format (from a link I found in StackOverflow) which was not easy reading let me just tell you. It boils down to this:

  1. Make a function using format which is a … library? … but I didn’t have to import it or anything. I think that’s because I link to the D3cdn? … at d3js.org. I like naming my variables and functions something descriptive like “formatMillions.”
  2. Then, use that function on your data inside a tickFormat function after the yScale you already have.

That was easily the worst job I have ever done explaining anything.

There seemed to be an endless list of other, far more complicated ways to do this but I found “.2s” (the winning answer at SO) to be pretty simple. It rounds your millions down to one decimal and adds an “M” to the end. I didn’t want one decimal place, I wanted two — because otherwise all my ticks look like they have the same value! Whoever asked the question followed up with “How do I get two decimal places?” but nobody answered. Several places on SO and elsewhere all mentioned “.2s” and gave other complicated ways to get two decimal places with things I didn’t want like the symbol for euros and emojis and whatnot. I kept looking in this pit of documentation and despair for a while before I finally just tried “.3s” and it worked.

Advertisements

Rock and Roll Hall of Fame Class of 2019 Inductee: Roxy Music

Part 2 in a series on Spotify and LastFM Data for the Rock and Roll Hall of Fame Class of 2019 inductees.

Personally, I don’t understand the whole Roxy Music thing. I only know one song by them and I wasn’t impressed. But, then, I know a lot of people don’t understand why The Stooges should have been in a lot sooner and Judas Priest should be in at all.

artistPage_RoxyMusic.png
Figure 01: Keep in mind that dip in February and subsequent climb back up was almost universal among artists on Spotify (because their popularity score is relative) so you have to ignore everything between early Feb to late March. All that to say, once you remove that, they have moved consistently upward.

I should also point out, in case you haven’t noticed, that this graph has a range of Roxy Music’s lowest popularity score to their highest (in this time period) and not 0-100 like the group charts in the first post of this series.  While using this range can be extremely useful to examine relatively small changes, it can be both shocking and confusing if you’re not aware of it.

Above the line chart, we’ve got their “profile pic,” if you could call it that, brought in via the Spotify Web API. Next to that, their current stats as of yesterday (for LastFM) and today for Spotify. I definitely need to beat Bootstrap into submission because it took me a couple hours to get those two elements as pretty as they are — you should have seen some of the horrors I brought to life playing with the CSS.

The Followers, Listeners, and Playcount numbers are a much more quantifiable and useful bunch. I’ll be doing even more cool things with those stats in the near future.

The uselessness of Spotify’s popularity score due to it’s relativity isn’t quite so ugly in when applied to charts like comparing albums to each other. In their defense, Spotify’s popularity score wasn’t designed for my convenience — it does exactly what it was meant for … creating the “top tracks” for any given category and so on.

albumsColumns_RoxyMusic.png

Albums often appear twice because they have more than one version — the initial release and a remastered edition, for example.

I can’t even recall reading anything explaining (not that they need to justify anything) to me what made them special or how they influenced others. The only lists — not that that’s necessarily evidence of anything — I’ve seen them on are those mentioning their Country Life album cover as being among the best and most controversial covers.

Using the Last.fm and MusicBrainz APIs

I am quite pleased with my app that uses the Spotify Web API for creating data visualizations using the popularity data of artists, albums, and tracks. That popularity score, however, is a relative number so I can’t truly compare much in a useful way, however pretty my UX is.

Now is the time on Sprockets where we start a new project using the Last.fm API to get real, quantifiable data — the number of listeners for each entity (artist, album, track) and playcount for each. I’m very interested to find how those numbers differ.

Spotify and the Spotify Web API make it very easy to get ID numbers for each artist, album IDs for each artist and track IDs for each album. In fact, the documentation shows they might even be making an effort to be stupid. The example URL in their documentation for getting info about an artist is:

/2.0/?method=artist.getinfo&artist=Cher&api_key=YOUR_API_KEY&format=json

So any spelling errors or similar names will make your life miserable. Fortunately, their database uses (for the most part–I’ll get to that in a moment) the mbid or MusicBrainz ID for everything so you can, instead, write the above like this:

/2.0/?method=artist.getinfo&mbid=bfcc6d75-a6a5-4bc6-8282-47aec8531818&api_key=YOUR_API_KEY&format=json

Unfortunately, there’s no Last.fm equivalent to Spotify‘s utterly delightful endpoint for getting all albums by an artist:

/artists/{id}/albums

Fortunately, the MusicBrainz API has such an animal but it’s a little more complicated and it took me a bit of trial and error to figure out quite how it works. I’d like to share it with you. Here are the steps I’ll need to complete to get all of the information about an artist, their albums, and the tracks from those albums with an extra curveball just to make it fun.

Step #1 – Search MusicBrainz.org for the desired artist

In this case, I chose Alice Cooper.

aliceCooperSearchResults.png

For the uninititated, not only would using the search term “alice cooper” bring up tribute bands but the fans (I count myself among them) who insist on differentiating between the group and the solo artist mean I have to use not one but two MBID numbers for this single artist.

Step #2 – Get the Artist’s MBID from MusicBrainz

Click the artist name in the search results then click the Details tab on the artist page.

aliceCooperDetailsTab.png

Step #3 – Get the Artist’s release-groups from MusicBrainz

Copy the artist MBID and paste it into a MusicBrainz web service URL using the artist entity and a ?inc=release-groups parameter like so:

/ws/2/artist/ee58c59f-8e7f-4430-b8ca-236c4d3745ae?inc=release-groups&fmt=json

to get a list of “release groups.”

AC_MB_info_browser.png

I broke up with Dreamweaver a while back, dumping it for the far smarter, cuter, cooler VS Code but ol’ DW will always have a special place in my heart because of cool commands like Edit > Code > Apply Source Formatting which does this:

beforeAfter.png

Remember, you have to do that twice for Alice Cooper. Because we, his fans, are pricks like that.

Step #4 – Get the Artist’s Releases from MusicBrainz

A release is an album — like Zipper Catches Skin or Billion Dollar Babies. A release group is all editions and versions of that album. To see exactly what that means, we copy the id for the Constrictor release-group from line 26 in the above image and construct the URL below using the release-group entity and inc=releases parameter.

/ws/2/release-group/24c4fdde-4494-3837-a288-5b4fdbe966eb?inc=releases&fmt=json

That gets us all six releases in the Constrictor release-group:

  • United States with “packaging”: “Cardboard/Paper Sleeve” (a vinyl record)
  • United States with “packaging”: “Jewel Case” (a CD)
  • Japan CD
  • United States with “packaging”: “” (digital?)
  • Germany with “packaging”: null
  • United Kingdom with “packaging”: null

I don’t know what null packaging means.

The first time I get data from Last.fm, I’ll try all of these and hypothesize “Cardboard/Paper Sleeve” will have a zero playcount.

Step #5 – Get a Release’s Recordings from MusicBrainz

A recording is a “track.”

Construct a URL using the individual release entity and inc=recordings parameter.

/ws/2/release/1189de93-7e9f-3b47-a9df-e2599e62f75d?inc=recordings&fmt=json

The UK release of Constrictor has 10 tracks. There will probably never be a deluxe or remastered edition. Which is a damn shame.

Step #6 – Get an Artist’s data from Last.fm

Using the artist MBID, method=artist.getinfo, and an API Key as parameters, assemble a Last.fm URL like so:

/2.0/?method=artist.getinfo&mbid=ee58c59f-8e7f-4430-b8ca-236c4d3745ae&api_key=YOURAPIKEY&format=json

The resulting JSON for Alice Cooper the man includes a bio, birthday, similar artists, genres, and–for me–the important bit:

"stats": {
"listeners": "1362113",
"playcount": "31011489"
}

You’re dying to know how that compares to Alice Cooper the group, amiright?

"stats": {
"listeners": "1362113",
"playcount": "31011489"
}

Well, check that shizzle out! We only need to run what will be this query once for Alice Cooper! Both MBIDs get the exact same data. Wanna know something nerdy and cool?

The artist entity URLs I used earlier to get the release groups differ …

  • Alice Cooper the person has a gender of “male.” Alice Cooper the band has a gender of “null.”
  • The person has a “begin_area” of Detroit, MI while the band has Pheonix, AZ
  • The band has a life-span of 1964-1975 with “ended”: true while the person’s life-span begins with “1948-02-04” and ends with “null” and “ended” is false.
  • Type properties are “group” and “person”

Step #7 – Get Album Info from Last.fm

We’ll go with Constrictor again. Using album.getinfo this time.

/2.0/?method=album.getinfo&mbid=1189de93-7e9f-3b47-a9df-e2599e62f75d&api_key=YOURAPIKEY&format=json

  • listeners: 64,771
  • playcount: 545,070

For comparison (the entire purpose of this whole exercise), let’s also get Billion Dollar Babies:

Well, I’ll be horn-swaggled! There were three Billion Dollar Babies release-groups in my JSON file but none of their IDs worked so I searched at MusicBrainz and got the actual mbid for the release-group which had 13 releases in it! How do you pick? Eventually, I’ll get all of them but …

First choice didn’t work … this is going to get tedious …

Okay, second try gets me some stats

  • listeners: 169,768
  • playcount: 1,222,112

Yeah, see that’s a significant difference.

Step #8 – Get Track Info from Last.fm

I went for “The World Needs Guts” from Constrictor using the track.getInfo method as a parameter.

/2.0/?method=track.getInfo&mbid=70bd458f-c62d-4014-99e7-233c3c0b8f96&api_key=YOURAPIKEY&format=json

  • listeners: 11,332
  • playcount: 38,656

Which I’m sure is tiny crumbs compared to even one instance of “School’s Out” let alone adding all of the recordings of “School’s Out” together.

Conclusion

It will take me a lot longer to get the data from Last.fm that I need than it does to get data from Spotify but it will also be data I can use!

Major Milestone Monday

After lots of hard work, I have D3 not only working with my database but have album art to go with the albums. It took me a while — even after I got the album covers to appear — to get the covers to display where I wanted them. I had to do math a lot. Still not “done,” but this is a big, big deal. This is my #BadAssTakeover.

I have cron jobs that regularly reach out to Spotify and store the popularity of artists, albums, and tracks. Here is a sampling of the bar graphs I can now do. Albums are in chronological order and the numbers are the popularity scores as of Sunday, April 22, 2018. Their popularity scores are relative to all albums on Spotify but I don’t know how they work … I don’t know if an album’s popularity takes into consideration the tracks’ popularity.

Click all images for full size

Alice Cooper

aliceCooper_01

Alice has tons of compilations which is frustrating for multiple reasons. He’s one of the artists that makes me think a vertical graph might be better.

aliceCooper_02

David Bowie

davidBowie_01

As you might expect, Bowie has a ton of albums even without compilations.

davidBowie_02

I’m not sure what the albums without art are … their titles are visible in a sortable table on a different page but I’m busy at the moment …

Bride

I chose them as a sample for a couple reasons. One, they have relatively few albums and two, they were played last night on IntenseRadio during The Great Scott Diversion.

bride

Deliverance

I really don’t like Deliverance but a lot of people do and they have a new album out.

deliverance

Iron Maiden

I feel like I should like Iron Maiden more than I do. Today was a bad Iron Maiden day for me. I’ve been trying to find a relatively inexpensive used copy of Live After Death (on vinyl) and finally found one at a local used record shop but they wouldn’t sell it to me because it wasn’t priced yet. How do I know it was relatively inexpensive if it wasn’t priced yet? The cover was in horrible condition.

ironMaiden

Led Zeppelin

I chose them because I wanted to show some really tall columns … all that “gray space” in the screen shots is how high up the columns could go.

ledZeppelin

Led Zeppelin is another reason my recent … I’ve started buying records again — for the first time in over 30 years — and it’s a lot less fun than I thought it would be.

Meat Loaf

meatLoaf

I think all those blank spaces are compilations. He’s like Alice that way. Ew.

Mortification

Mortification, like Deliverance, are in my database because I wanted a health selection of thrash and black metal and similar artists to compare to each other.

mortification

One Bad Pig

Because I love them.

oneBadPig

They only have four albums which is sad. Well, they’ve also got a demo and a live album but those aren’t on Spotify.

Resurrection Band/Rez Band/Rez/Resurrection Band

rez

For some reason, there’s at least one … wait … it does say Resurrection Band … I thought Ampendectomy was a Glenn Kaiser solo album. Can you believe Lament isn’t their most popular album? People are dumb.

You Have No Idea

There is no greater technological accomplishment in the history of man:

yay

Not the printing press, not the plow, not radio, penicillin, birth control pills, or the computer compare to this. I am king of the world.

Dream Job Identified

First, I have to mention a job I saw on Indeed the other day: Data Analyst for the Tampa Bay Rays. Among the many bulleted job requirements was, “a rudimentary understanding of baseball” as if they knew oodles of nerds would apply that couldn’t tell an interviewer how many balls resulted in a walk. As for me, if they asked me about baseball, I could at least tell them Moneyball was one of my all-time favorite movies.

All kidding aside, I miss going to Tigers games a lot but I’d be so uninterested in working for a sports team. As fascinating as I find PornHub‘s Insights blog, I’d also be reluctant–to say the least–to work there.

I’ve long fantasized, however, about having access to Netflix‘s data. Much like the PornHub stats, I’m fascinated by what people in different demographics watch, do, and think and how that changes at certain times of day. I love reading about album sales and get so frustrated that some albums’ sales aren’t tracked anymore. I love the idea that someone’s commute can reveal so much about them and their whole lives. I love the relationships between things. I love Freakonomics.

I love data.

While poking around Spotify‘s resources, I saw a link following the question, “Love Python and want to use it at Spotify?” so I took a look at their list of openings. Of all the wickedly cool sounding data-related positions they had, “Apps & Visualization Designer” seemed the most like, well, my destiny. The bullets beneath the “Who You Are” heading was like a list of qualities … like someone was describing me. I’d be like Jonah Hill in Moneyball.

Is This 1986 or 2016?

At 46-years-old, I’m still doing my homework while listening to Twisted Sister.

eh.jpg

In my 1986/87 dorm room at Howe Military School, I’d crank Come Out and Play on Friday and Saturday nights while cleaning for the next morning’s inspection.

Love Is for Suckers is one of the most underrated albums of all time. I’d just registered for my first (and last) semester at The University of Michigan when I rode my bike to the record store to buy the album & ticket with my minimum wage ($3.35/hr) earnings from Rent-a-Flick Video. Fortunately, I had my first car by the time the concert came around.

They had to cancel that tour because only myself and three or four other people went. While Dee Snider was high-fiving people, he touched the very tip of one of my fingers and I haven’t washed it in thirty years.

After searching for years for a setlist or even a confirmed date, I finally found them both this morning and made a Spotify playlist to relive that glorious evening.