Spotify and LastFM Data for Rock and Roll Hall of Fame Class of 2019

The ongoing project I’m usually busiest with is PopRock — a web app that tracks statistics and creates data visualizations using data from the Spotify Web API and the Last.fm API.

From Spotify, I grab artist Popularity and Followers (daily) as well as Popularity for albums and tracks (weekly). That is

  • 327 artists
  • Their 5,742 albums
  • The 75,283 tracks from those albums

If you’ve never heard of Last.fm, it gathers data via “scrobbles” from all participants who have a “scrobbling” app installed. When you, the music lover, listen to a song using iTunes, Spotify, etc. the app sends that bit of information as a “scrobble” to Last.fm. As often as I remember to, I run a Python script that gathers the Listeners and Playcount for each artist, their albums and tracks. At the moment, this task is so cumbersome, I keep the list relatively short — 32 artists.

What brings joy to my nerdy little heart is when there’s an event that creates an anomaly or jump in my data. For example, a movie about Queen causes a spike in popularity — especially for a particular song (“Bohemian Rhapsody”) or a movie about Motley Crue causes a spike in their popularity.

What brings sadness to my heart is how little influence the Rock and Roll Hall of Fame has on my little data exercises. For the most part, neither inductions nor nominations cause much of a change. Having said that, I look forward to the announcement of nominations every year as well as to the broadcast of the induction ceremony. The ceremony was held last week and the ceremony will air on HBO April 27. To celebrate that, I’m going to share some data and charts for this year’s nominees and inductees.

Below (Figure 01) is the Spotify popularity for the Class of 2019.

compareBars.png
Figure 01: L-to-R Def Leppard, Janet Jackson, Radiohead, Roxy Music, Stevie Nicks, the Cure, the Zombies

I’ve already written about how Spotify calculates their popularity score numerous times and every article about the RRHoF repeats their rules and controversy so forgive me if I don’t regurgitate all of that here.

Induction Process and Eligibility Requirements, etc.

compareLines_Inductees.png
Figure 02: A line starts on the day (or day after) I add an artist to my database.

As you can see, all artists flow together for the most part — because Spotify’s popularity scores are relative. That’s what caused all of them to go down together in late June, climb back up and fall again together in February. Given that, the lines, in a vacuum, would be pretty straight. Knowing that, the nominations announced in December 2018 seem to have no effect.

Speaking of nominations, here are those who were nominated last year but not inducted.

compareBars_Noms.png
Figure 03: L-to-R: Devo, John Prine, Kraftwerk, LL Cool J, MC5, Rage Against the Machine, Rufus featuring Chaka Khan, Todd Rundgren

Like many people, I find it extremely difficult to keep my thoughts about this to myself. At least three of those artists deserve to be in more than at least three of the inductees. But, of course, if the RRHoF used reason and logic, people wouldn’t write motherloads of articles and blog posts about them twice a year.

compareLines_Noms.png
Figure 04: Yes, I know I need to tweak a couple things. It’s a work in progress.

The only artist I would say was really affected by their nomination is the MC5. I would expect all of them would bump up because the public was reminded of these artists and listened to them like, “Oh, yeah, I haven’t listened to LL Cool J in forever” but, in the case of the MC5, I think it’s more like, “Who are the MC5?” and I am just fine with a new generation being exposed to them by any means necessary. Very similar to another highly-influential Detroit band, The Stooges (inducted 2010). I’m still waiting for Ted Nugent to get selected for consideration to be put on the ballot. Once.

Because popularity is relative, you might be thinking the line charts are pretty useless and you’d be right. If I look at each artist individually, however,  and change the Y-axis from a 0-to-100 scale to a lowest-to-highest scale, they become (a little) more useful. Sometimes. This begins a series of posts about each inductee that do exactly that with some more useful statistics and data visualizations.

Before we go on, however, here’s something more useful — a graph using each artist’s followers on Spotify.

followerColumnsX.png
Figure 05: L-to-R: Def Leppard, Kraftwerk, Janet JacksonJohn Prine, DevoLL Cool J, MC5, RadioheadRage Against the Machine, Roxy MusicRufus featuring Chaka Khan, Stevie Nicks, The Cure, The ZombiesTodd Rundgren

Day-um! Radiohead has some fans, yo!

That wide range between, for example, The Zombies (who deserve better, by the way) and Radiohead makes for what could be considered a clunky chart.

In other charts, the ratio for popularity (which is, conveniently, on a scale of 0-100) is 1:2. One popularity point gets represented by two pixels so I can fight against the opposite problem of a chart that’s too small. The ratio in the above chart is 5000:1. 5000 followers to one pixel.

Chrome’s dev tools were invaluable in troubleshooting that.

I love how “good” I’m getting at math now that it’s always in a real, practical context. For the labels on top of the columns I first checked to see whether the number was above or below one million.

  • If it was below, I divided by 1000 and used Javascript‘s toFixed(0) method to strip away the decimal places — which super conveniently converts the integer to a string which I concatenated with a “k”.
  • If it was above, I divided by one million, used toFixed(1) and added the “M”.
Advertisements

Same Artist with Different MBIDs: MusicBrainz, LastFM (and Spotify)

TL;DR ==> Skip the Introduction and scroll to “The Actual Tests”

Introduction

One of the cooler (to me) features of my PopRock app is it combines related artists for some of the charts–if the same artist is in multiple bands, for example, like Ronnie James Dio was in Dio, ElfBlack Sabbath, Rainbow, and so on. That was fun to work on and most triumphant when I finally got it to work.

MusicBrainz and LastFM, however, have provided me with a somewhat similar challenge. In my first post about using the MusicBrainz and LastFM APIs, I mentioned that Alice Cooper has two MBIDs–one for the band (1964-1975 according to MusicBrainz) and another for the lead singer cum solo artist (or is it “solo artist née lead singer”?). I’ve stored those two MBIDs, along with several others, in a python list that I loop through to get data about multiple artists. I wasn’t yet ready to store this data in my database, so I was saving each artist’s daily data in a JSON file (which Python makes so freaking easy, by the way!) like so:

artistNameFor_file_name = artistName.replace(‘ ‘, ”)

dateFor_file_name = time.strftime(“%m-%d-%y”)

artistJSON = json.dumps(artist, indent=4)

f = open (‘data/’ + artistNameFor_file_name + ‘_‘ + dateFor_file_name + ‘.json‘, ‘w’)

f.write (artistJSON)

f.close()

It works totally awesome — even for artists with unusual characters in their name such as Mötley Crüe. It doesn’t work so well for artists with exactly the same name such as Alice Cooper and Alice Cooper. In the latter case, data for the second Alice Cooper overwrites the file and data for the first Alice Cooper.

I could use the ‘a’ (instead of the ‘w’) argument to append it but for it to merge the data how I want it* isn’t really worth the time and effort it would take to write that script. What I’ll do, instead, is write the PHP for putting both sets of data where it belongs in my MySQL database which I have to do anyway. What I noticed while writing that previous post was LastFM isn’t a pretentious wiener like the guys who work for Championship Vinyl in High Fidelity (and MusicBrainz contributors). LastFM uses only one of those MBIDs for  Alice Cooper. I just have to test both MBIDs again to see which one — then I can write … wait … I could add some … other string from an artist’s data to the filename … OR I could include the time in addition to the date … and that might be helpful for … something else as well. But, as I said, I need this PHP written as well.

Back to the primary topic:

I need a script (and maybe a … are they called “lookup” tables?) that knows to put data from both Alice Cooper MBID … crap … no, wait … because these are hard, quantifiable (yes, I know I’m mis-using that word … be thankful I didn’t say “mis-abusing”) numbers I can add them! That’s the whole purpose behind getting LastFM data in the first place!

I am hoping Joan Jett is equally easy. For some reason, MusicBrainz contributors have her under Joan Jett and Joan Jett and the Blackhearts. I have no idea why. She’s never done anything without the Blackhearts. This is not a Tom Petty and Tom Petty & the Heartbreakers situation … aw, man! … I also need to deal with Tom Petty.

Spotify has one artist id for all Alice Cooper albums as well as Joan Jett. They have different, separate ids for the two Tom Petty entities which I’m okay with (and I’m sure Spotify is relieved). There are other issues with Spotify both odd but whatever as well as inaccurate but … that’s a whole different, well, issue.

The Actual Tests

TEST #1 – Part 1

First, I’ll demonstrate the “problem” by showing the response to a request to MusicBrainz for the Alice Cooper group’s Release Groups (a list of albums containing all the different releases–like other countries, etc.–of that album).

ACgroup_MB_ReleaseGroups.png
The Alice Cooper band MBID data includes 25 release groups (for only 7 “real life” albums! Still only 8 if you include Greatest Hits).

OMG! I just noticed there is a “type” property with a “Group” value! Woo-hoo! That is going to make everything sooooo easy.

ACperson_MB_ReleaseGroups.png
MusicBrainz has 25 release groups for Alice Cooper as a solo artist. Exactly the same amount as the group which is odd considering he has 20 studio albums (which doesn’t include live albums or compilations). That … is weird.

I notice, with a smile, this MBID has a “type” of “person” key/value pair.

I have to note that the Alice Cooper discography Wikipedia page combines solo artist and group lists for studio albums, live albums, and compilations but separates group and solo artist singles. Alice Cooper is just a mess. 😉

Important part: Each MBID has 25 albums for a total of 50. Yes, I read all of that and know each MBID has a different list of albums.

TEST #1 – Part 2

Artist Info from LastFM using the person vs band MBIDs

ACcombined_LastFM_artistInfo.png

The LastFM response using the person MBID gets me artist info for the same MBID.

The LastFM response using the group MBID redirects me to the artist info for the person MBID and the data is the same — note the listeners and playcount values.

Moral: I only need to use one MBID for getting info from LastFM — the person MBID.

Let’s try Joan Jett.

JJperson_MB_ReleaseGroups.png
MusicBrainz has 9 release groups for Joan as an individual.

I’m amused they have hometowns (“begin_area”) for both Alice (Detroit) and Joan (Wynnewood) but for the Alice Cooper group (Pheonix) and JJ & the Blackhearts (Los Angeles) they have the city in which the band was formed.

JJperson_MB_ReleaseGroups.png
MusicBrainz has 25 release groups for Joan Jett and the Blackhearts!

Joan Jett & Company isn’t quite as simple with LastFM.

JJperson_LastFM_artistInfo.png
Joan Jett individual MBID stats from LastFM

Joan Jett the person is, apparently, similar to neither the artist who inspired everything about her (Suzi Quatro) nor her contemporary, Pat Benatar.  Okay.

JJband_LastFM_artistInfo.png
Joan Jett and the Blackhearts MBID stats from LastFM

JJ & the BHs are, unsurprisingly, similar to her old band The Runaways and, well, Lita Ford (lead guitarist for The Runaways).

Neither “solo” or with the BHs (and those should be, in truth, the same MBID) is similar to Evil Stig (a band for which JJ sang 100% the lead vocals, not just some as she did with the Runaways).

Most importantly and irritatingly, the MBIDs have different values for listeners and playcount. I can still simply add them but this requires me to fetch each MBID in Last FM separately (for the time being, while I’m using my Python script to put the responses into JSON files).

Update: I’ve made a little lookup-ish table.

queryForMBID.png

I know I don’t need the name columns, but it helps me. This is for artists that are identical in the real world to make sure data from LastFM goes where it should — Alice Cooper has all Alice Cooper stats, albums, and tracks and Joan Jett gets all her stuff.

Speaking of Joan Jett, I am about to–any second now–make a lookup-ish table for related artists so I don’t need my “manual” group charts below.

steveTaylorGroup.png
This chart uses arrays containing an artist and groups for which they were the only lead singer.

On a related note, I’ll make a “related albums” table for artists who were only on some of a band’s albums — for example, any Dio-related query knows to get only those Rainbow and/or Black Sabbath albums for which he was the lead singer.

An issue that will still remain is albums for which an artist is only on one or some of the songs:

  • My albums table won’t accept albums if the artistID exists in the artist table which means neither The Rocky Horror Show nor The Rocky Horror Picture Show get inserted because they’re cast albums with various singers–Meat Loaf‘s artistID won’t match the artistID for those albums and “Various” or whatever isn’t in the artists table.
  • Ted Nugent‘s Free for All album doesn’t display for Meat Loaf queries despite Loaf singing some of the songs — he isn’t the “artist” for the album. Hmm … I should look and see if he shows up in any of properties for that album in any of my data sources … my goodness but this app is becoming wicked cool wicked fast!

Footnotes

*Once I increase my MongoDB and CouchDB knowledge, perhaps it won’t be such a time-consuming task.

Using the Last.fm and MusicBrainz APIs

I am quite pleased with my app that uses the Spotify Web API for creating data visualizations using the popularity data of artists, albums, and tracks. That popularity score, however, is a relative number so I can’t truly compare much in a useful way, however pretty my UX is.

Now is the time on Sprockets where we start a new project using the Last.fm API to get real, quantifiable data — the number of listeners for each entity (artist, album, track) and playcount for each. I’m very interested to find how those numbers differ.

Spotify and the Spotify Web API make it very easy to get ID numbers for each artist, album IDs for each artist and track IDs for each album. In fact, the documentation shows they might even be making an effort to be stupid. The example URL in their documentation for getting info about an artist is:

/2.0/?method=artist.getinfo&artist=Cher&api_key=YOUR_API_KEY&format=json

So any spelling errors or similar names will make your life miserable. Fortunately, their database uses (for the most part–I’ll get to that in a moment) the mbid or MusicBrainz ID for everything so you can, instead, write the above like this:

/2.0/?method=artist.getinfo&mbid=bfcc6d75-a6a5-4bc6-8282-47aec8531818&api_key=YOUR_API_KEY&format=json

Unfortunately, there’s no Last.fm equivalent to Spotify‘s utterly delightful endpoint for getting all albums by an artist:

/artists/{id}/albums

Fortunately, the MusicBrainz API has such an animal but it’s a little more complicated and it took me a bit of trial and error to figure out quite how it works. I’d like to share it with you. Here are the steps I’ll need to complete to get all of the information about an artist, their albums, and the tracks from those albums with an extra curveball just to make it fun.

Step #1 – Search MusicBrainz.org for the desired artist

In this case, I chose Alice Cooper.

aliceCooperSearchResults.png

For the uninititated, not only would using the search term “alice cooper” bring up tribute bands but the fans (I count myself among them) who insist on differentiating between the group and the solo artist mean I have to use not one but two MBID numbers for this single artist.

Step #2 – Get the Artist’s MBID from MusicBrainz

Click the artist name in the search results then click the Details tab on the artist page.

aliceCooperDetailsTab.png

Step #3 – Get the Artist’s release-groups from MusicBrainz

Copy the artist MBID and paste it into a MusicBrainz web service URL using the artist entity and a ?inc=release-groups parameter like so:

/ws/2/artist/ee58c59f-8e7f-4430-b8ca-236c4d3745ae?inc=release-groups&fmt=json

to get a list of “release groups.”

AC_MB_info_browser.png

I broke up with Dreamweaver a while back, dumping it for the far smarter, cuter, cooler VS Code but ol’ DW will always have a special place in my heart because of cool commands like Edit > Code > Apply Source Formatting which does this:

beforeAfter.png

Remember, you have to do that twice for Alice Cooper. Because we, his fans, are pricks like that.

Step #4 – Get the Artist’s Releases from MusicBrainz

A release is an album — like Zipper Catches Skin or Billion Dollar Babies. A release group is all editions and versions of that album. To see exactly what that means, we copy the id for the Constrictor release-group from line 26 in the above image and construct the URL below using the release-group entity and inc=releases parameter.

/ws/2/release-group/24c4fdde-4494-3837-a288-5b4fdbe966eb?inc=releases&fmt=json

That gets us all six releases in the Constrictor release-group:

  • United States with “packaging”: “Cardboard/Paper Sleeve” (a vinyl record)
  • United States with “packaging”: “Jewel Case” (a CD)
  • Japan CD
  • United States with “packaging”: “” (digital?)
  • Germany with “packaging”: null
  • United Kingdom with “packaging”: null

I don’t know what null packaging means.

The first time I get data from Last.fm, I’ll try all of these and hypothesize “Cardboard/Paper Sleeve” will have a zero playcount.

Step #5 – Get a Release’s Recordings from MusicBrainz

A recording is a “track.”

Construct a URL using the individual release entity and inc=recordings parameter.

/ws/2/release/1189de93-7e9f-3b47-a9df-e2599e62f75d?inc=recordings&fmt=json

The UK release of Constrictor has 10 tracks. There will probably never be a deluxe or remastered edition. Which is a damn shame.

Step #6 – Get an Artist’s data from Last.fm

Using the artist MBID, method=artist.getinfo, and an API Key as parameters, assemble a Last.fm URL like so:

/2.0/?method=artist.getinfo&mbid=ee58c59f-8e7f-4430-b8ca-236c4d3745ae&api_key=YOURAPIKEY&format=json

The resulting JSON for Alice Cooper the man includes a bio, birthday, similar artists, genres, and–for me–the important bit:

"stats": {
"listeners": "1362113",
"playcount": "31011489"
}

You’re dying to know how that compares to Alice Cooper the group, amiright?

"stats": {
"listeners": "1362113",
"playcount": "31011489"
}

Well, check that shizzle out! We only need to run what will be this query once for Alice Cooper! Both MBIDs get the exact same data. Wanna know something nerdy and cool?

The artist entity URLs I used earlier to get the release groups differ …

  • Alice Cooper the person has a gender of “male.” Alice Cooper the band has a gender of “null.”
  • The person has a “begin_area” of Detroit, MI while the band has Pheonix, AZ
  • The band has a life-span of 1964-1975 with “ended”: true while the person’s life-span begins with “1948-02-04” and ends with “null” and “ended” is false.
  • Type properties are “group” and “person”

Step #7 – Get Album Info from Last.fm

We’ll go with Constrictor again. Using album.getinfo this time.

/2.0/?method=album.getinfo&mbid=1189de93-7e9f-3b47-a9df-e2599e62f75d&api_key=YOURAPIKEY&format=json

  • listeners: 64,771
  • playcount: 545,070

For comparison (the entire purpose of this whole exercise), let’s also get Billion Dollar Babies:

Well, I’ll be horn-swaggled! There were three Billion Dollar Babies release-groups in my JSON file but none of their IDs worked so I searched at MusicBrainz and got the actual mbid for the release-group which had 13 releases in it! How do you pick? Eventually, I’ll get all of them but …

First choice didn’t work … this is going to get tedious …

Okay, second try gets me some stats

  • listeners: 169,768
  • playcount: 1,222,112

Yeah, see that’s a significant difference.

Step #8 – Get Track Info from Last.fm

I went for “The World Needs Guts” from Constrictor using the track.getInfo method as a parameter.

/2.0/?method=track.getInfo&mbid=70bd458f-c62d-4014-99e7-233c3c0b8f96&api_key=YOURAPIKEY&format=json

  • listeners: 11,332
  • playcount: 38,656

Which I’m sure is tiny crumbs compared to even one instance of “School’s Out” let alone adding all of the recordings of “School’s Out” together.

Conclusion

It will take me a lot longer to get the data from Last.fm that I need than it does to get data from Spotify but it will also be data I can use!

Rock and Roll Hall of Fame 2018 Nominees

The latest nominees were announced in the last couple of days so I was in a sudden rush today to fix a bunch of stuff so I could get them into my database. Hence, I used the old, icky method. I’m not bummed out about that, though, because I did get a ton of stuff fixed that had become broken over time — including changing from PHP session variables to PHP cookies. I also fixed all the column-sorting links and related page-building functions that were jacked up. So, I can take screenshots of some the latest top tracks and albums for this years nominees.

Spotify, because they hate people, doesn’t make the number of plays available through their API — also not available in their web player. “Popularity” is relative to other artists, albums, and tracks on Spotify so the number may go up and down from day to day. So, from what I can guess, an artist’s top albums or tracks may not really be the top tracks or albums by that artist — because the numbers are calculated relative to all other albums and tracks — not like you’re ranking an artist’s songs by number of plays.

Yes, it’s stupid. It’s not just you.

Even more stupid is Last.fm‘s API which totally got my hopes up because you can access the number of plays! But … not the identifiers for the artists, tracks, or albums.

Because they’re stupid. That’s why.

But, in the words of Count Rogin, “Let’s work with what we have, shall we?”

Janet Jackson

janetTracks.png

janetAlbums

WordPress is ridiculously stupid. Depending on the browser and OS you’re using, the tools change. Sometimes you can control how large an image is when it’s placed. Sometimes, you can also make it so a user/reader can click it for a larger version. That seems to be increasingly rare so I apologize for how difficult to read some of these charts are.

John Prine

prineTracks.png

Each column can be sorted in the app by clicking on “Album Name”, “Track”, or “Track Popularity”.

prineAlbums.png

Again, I apologize. When an artist has a ton of albums, it makes for lots of horizontal scrolling so, at some point, I’ll make the bars listed vertically so it doesn’t suck even on a desktop.

Roxy Music

roxyTracks.png

roxyAlbums.png

Eventually, I’ll have the years appear beneath the albums. The albums, by the way, are ordered chronologically by release date.

Stevie Nicks

nicksTracks.png

I included more tracks by Nicks because I thought it was interesting that “Rooms On Fire” made it into the top five while “Talk To Me”, the other single from Rock A Little was at #23.

nicksAlbums.png

Todd Rundgren

rundgrenTracks.png

I grabbed the top 11 tracks because, again, I was rather amused by the fact that four of the top 11 were all “Bang on the Drum All Day.”

I am currently working on an algorithm that would somehow combine all the scores for the same song across albums (including remasters and compilations but not live versions). Same for albums … I’d like a score that took into account several versions of the same album.

At some point, I realized I could take screenshots of the app’s “list” view for albums. In list view, I can sort them, just like the list view for tracks.

rundgrenAlbums.png

Def Leppard

defTracks.png

I grabbed so many more of theirs because, well, I cared more.

defAlbums.png

See how there’s two versions of so many of their albums? I find that irritating. It lowers the tide for all of the albums and tracks in question.

Devo

devoTracks2.png

More rows of Devo because, again, I cared more.

devoAlbums.png

 

Kraftwerk

kraftTracks.png

kraftwerkAlbums.png

I like it when there are fewer albums. The screenshots are prettier.

LL Cool J

coolJtracks.png

coolJalbums.png

MC5

mc5tracks.png

Not a big surprise there for the #1 track.

mc5albums.png

Radiohead

radioTracks.png

radioHeadalbums.png

Rage Against the Machine

rageTracks.png

rageAlbums.png

Rufus

I know for a fact I didn’t add Rufus to my database until they were nominated last year so this line graph represents their popularity over the last year. You’re welcome.

rufusLine.png

rufusTracks.png

rufusAlbums.png

The Cure

cureTracks.png

cureAlbums2.png

The Zombies

The Zombies are nominated again which makes me happy. I think they stand a better chance of getting in this year.

zombiesTracks.png

No surprise in that list. I think what might hurt them is many people might be like me and can only name those top two tracks.

zombiesAlbums2.png

I love that Still Got That Hunger is so high on their top albums list. It deserves it. Good on ya’, guys!

“More Popular Than Jesus” – John Lennon

Hopefully, the post title won’t get me the same backlash it got Lennon.

I am currently tracking approximately 146 artists using the Spotify Web API using the popularity endpoint for artists, albums, and tracks to do ridiculously interesting things like write this blog post and, eventually, soon … do crazy cool interactive data visualization stuff.

topTen
The Top 10 (of 146)

Something I noticed immediate upon sorting all of them according to popularity is this: Christian artists are the least popular. Not like on average but … they’re all at the bottom.

bottom.png
The Bottom 28 (of 146) contains only one “secular” artist.

If you don’t count Johnny Cash, Bob Dylan, or Alice Cooper as “Christian,” I have 42 Christian artists.

Of the bottom 50 artists, 40 of them are Christian artists.

The highest Christian artist is DC Talk with a popularity score of 48.

Then, down below Sodom and Gwar, Stryper shows up with 46 popularity.

Just below Stryper is Venom with 46.

To say Slayer is #50 on the list (with 67 popularity) is a bit misleading because many of the artists have the same popularity score but I thought it worth mentioning.

I’m sure there are many artists “below” them in Spotify’s massive database and, if I’m honest, it makes sense that such niche artists would fall below much more mainstream artists but … it still struck me as … odd … and … sad.

APIs I Need

There are some sites I would love to beat into submission–eBay and Craigslist, for example. I’ve just this very day fallen in love with discogs.com but for every record I want to add to my Wantlist, I have to filter a whole bunch of crap and that got tedious real quick. I’d also like to shop and cross-reference some shizzle … for example, I’m sure everyone has been trying to buy multiple used items and wished there was an easy way to find out which vendors had two or more of the items you want … so I need to make that shizzle. Fortunately, there are some existing APIs and some neat stuff I found on GitHub I am adding to my To-Do pile.

eBay

https://go.developer.ebay.com/

https://go.developer.ebay.com/api-documentation

Discogs

https://www.discogs.com/developers/

CraigsList

https://github.com/juliomalegria/python-craigslist

https://gist.github.com/flodel/2573531–

https://www.npmjs.com/package/node-craigslist

http://ghostproxies.com/blog/2015/09/the-ultimate-guide-to-scraping-craigslist-data-with-software/

Duuuuuuude!

Again, I know this is no big deal but I’ve never had such an uninterrupted stretch of things working as expected. I don’t mind the unexpected or things not working as long as I don’t spend days and days trying to figure out why. I like mistakes. I like fixing things. I like a learning curve.

Behold, my array.

array.png
Including compilations returns 42 items in the array.

I was pleasantly surprised at that little array panel opening on the right. AND the items are numbered!

Now to do that a few more times and replacing the old functions because …

I was making a motherload of requests (resulting in 423 “too many requests” errors) because, depending on how you request and album or song, you may or may not get all the properties you want. For example, Get An Artist’s Albums doesn’t give you each album’s popularity and Get An Album’s Tracks doesn’t give you each track’s popularity. Get An Album and Get A Track do. As you can see in the image above, depending on how you make the request, Alice has 37-67 albums and some of those albums are big-ass box sets with dozens of tracks.

I was pleased to discover I could reduce the number of requests by storing all albumIDs in an array and making a request using Get Several Albums (which, as it turns out, gives me Popularity I want and need) but, while Get An Artist’s Albums has a maximum limit of 50, Get Several Albums has a maximum limit of 20.

And so I think my pseudo code outline currently looks like this:

  1. Build artistURL using artistID
  2. Use artistURL to request artist JSON
    • Get artist name
    • Get artist popularity
  3. Build artistAlbumsURL using artistID
  4. Use artistAlbumsURL to request artist’s albums JSON
    • For each album
      • Push albumID into artistAlbums array
  5. Check for length of artistAlbums array
    • Splice into arrays of 20 items or less
    • For each array of artistAlbums
      • Request Several Albums
        • For each album
          • Get album popularity
          • Get album release date
          • Push trackIDs into albumTracks array
          • Check for length of albumTracks array
            • Splice into arrays of 50 items or less
            • For each array of albumTracks
              • Request Several Tracks
                • For each track
                  • Get track name
                  • Get track popularity