Finding and Deleting (almost) Duplicate Rows in MySQL

The database for my PopRock web app has evolved as I learn and as my needs & wants expand. Learning means mistakes and mistakes with MySQL (and PHP) mean duplicate rows. Sometimes lots of them–I once forgot to empty an array at the end of a loop (inside another loop). It didn’t take long to notice that page was getting increasingly slower … because it had millions of rows.

The PopRock database contains data fetched from the Spotify Web API using cron jobs for:

  • Artist popularity – daily
  • Album popularity – weekly
  • Track popularity – weekly

In the last couple days, I noticed some of these items hadn’t been fetched in a couple months–or a couple years.

Sidebar/Note to self: I also noticed that some albums (anthologies) didn’t have all their tracks in my database — my solution to (and explanation of) that problem will probably be my next post.

Debugging and testing also has the side effect of creating lots of duplicate rows. For example:

albumID_01.png
Figure 01

There’s also this from when I added a date column after I’d started fetching data. I am fine with deleting all of these and doing so is easy.

noDateInPopTracks.png
Figure 02

And, lastly, for a bit, my cron job was getting twice a day:

twice.png
Figure 03

This happened because my albums and tracks cron jobs works like this:

  1. Get an array of artists
  2. For each artist
  3. get an array of their albums
  4. For each album, get that album’s popularity score from Spotify

I do have one array containing all of my artists for getting their popularity scores but that task is relatively small. Looping through every artist then getting all of their albums is a bigger task. Then getting all of the tracks for each album is even bigger. So, for those last two tasks, I’ve broken up my artists into smaller arrays.

If, accidentally, I’ve placed one of my artists into two of those arrays, everything for that artist gets done twice (that last screenshot). How does one accidentally get millions of rows like I mentioned earlier? One accidentally writes (or copy & pastes together) a script that, basically, says something like: For every track in David Bowie’s oeuvre, loop through all of David Bowie‘s albums that loop through every track on each of those albums. Yeah, imagine that grows like a zombie apocalypse.

My page for listing all an artist’s tracks has been slowing again, lately, so I need to:

  • Find out if something works but is happening more than it should
  • Resume work on making PopRock a Progressive Web App (PWA) so it stores data locally, speeding up everything it does.

Finding & Deleting Duplicate Rows

Finding and deleting all “duplicate” rows for Figure 02 is easy. I just need to find all rows where date equals 0000-00-00 00:00:00.

SELECT * FROM popTracks WHERE date='0000-00-00 00:00:00';

That query returns 1,116 rows out of 1,622,599 for (currently) 72,997 tracks so I’m confident I won’t lose anything significant when I delete them like

DELETE FROM popTracks WHERE date='0000-00-00 00:00:00';

leaving 1,621,483.

The other two aren’t so easy.

I don’t want to delete every row inserted on a given date–I want to delete all but one. Also, I created an extra hurdle when coding my PHP. When I get a new album to insert into my albums table, I format the date:

$albumReleasedWhole = $album->release_date;
$albumReleased = substr($albumReleasedWhole, 0, 4);

I do that because some release_dates are a four digit year, some are month and year, and some are full dates. I didn’t do that for the popAlbums table — I didn’t anticipate ever caring about the date format.

So I have dates that are kind of alike. In Figure 01, I have sixteen rows inserted on March 10 but all at different times. If I muster my limitless optimism and try using just the date …

SELECT * FROM popAlbums WHERE date='2018-03-10';

that fails. Zero rows.

We know that if I were to select those dates to display on a web page, I can write something like the above PHP so what I’m going to do is

  1. Create a new column called, say, justDate.
  2. Fill all the empty cells in the justDate column with a converted date from the old date column.
  3. I’ll scour all those rows for any obvious yuckiness and test for any less-obvious.
  4. Once I’m sure everything is okay, I can delete the old date column.

I created the new column under the Structure tab in PHPMyAdmin, then hopped over to the SQL tab and:

UPDATE popAlbums SET justDate = date_format((popAlbums.date), '%Y-%m-%d');

Ta-da!

Figure 04

Because I’m a very cautious man, I copied the table before trying the next couple things without reducing myself to a sobbing mess on the floor. In popAlbums2, I deleted the original date column and proceeded with this:

DELETE a FROM popAlbums2 a
INNER JOIN popAlbums2 b
WHERE a.albumID = b.albumID AND a.justDate = b.justDate AND a.id < b.id;

In line 2, we create two tables (sort of–if we make a change in a, it also happens in b because they’re the same table–don’t think about it too hard) by using INNER JOIN.

Line three looks at all the rows in b. When it finds more than one row for a given album on a given day (a.albumID = b.albumID and a.justDate = b.justDate) like the sixteen rows on March 10 in Figure 01, it compares the id of the row in b to the id of the first similar row in a:

  • Is a.id ‘6001’ less than b.id ‘6001’? No. Move on to the next row in b.
  • Is a.id ‘6001’ less than b.id ‘6037’? Yes. Delete it and move on.

I held my breath and ran the query for a very, very long 16 seconds.

Rows affected = 11,850

I am SO glad I made a copy of this table. I moved back over to the Browse tab and … here’s the final result!

Figure 05 (Compare to Figure 01)

Lastly, I renamed the popAlbums table to popAlbumsOLD and renamed popAlbums2 to popAlbums as well as renamed the justDate column to date so all my functions and pages still work.

I had to hold my breath for a much longer 33 seconds while I ran the date_format query on the popTracks table’s 1,646,115 rows.

My DELETE query put me through, as NASA would say, 187 seconds of terror (it seemed like a lot longer) to delete 48,492 rows.

UPDATE: Fortunately, I remembered I had to change how the date gets inserted into the new column. Before, the default was just “current timestamp.” In my PHP, I changed it to use the MySQL curdate() function.

Advertisements

Same Artist with Different MBIDs: MusicBrainz, LastFM (and Spotify)

TL;DR ==> Skip the Introduction and scroll to “The Actual Tests”

Introduction

One of the cooler (to me) features of my PopRock app is it combines related artists for some of the charts–if the same artist is in multiple bands, for example, like Ronnie James Dio was in Dio, ElfBlack Sabbath, Rainbow, and so on. That was fun to work on and most triumphant when I finally got it to work.

MusicBrainz and LastFM, however, have provided me with a somewhat similar challenge. In my first post about using the MusicBrainz and LastFM APIs, I mentioned that Alice Cooper has two MBIDs–one for the band (1964-1975 according to MusicBrainz) and another for the lead singer cum solo artist (or is it “solo artist née lead singer”?). I’ve stored those two MBIDs, along with several others, in a python list that I loop through to get data about multiple artists. I wasn’t yet ready to store this data in my database, so I was saving each artist’s daily data in a JSON file (which Python makes so freaking easy, by the way!) like so:

artistNameFor_file_name = artistName.replace(‘ ‘, ”)

dateFor_file_name = time.strftime(“%m-%d-%y”)

artistJSON = json.dumps(artist, indent=4)

f = open (‘data/’ + artistNameFor_file_name + ‘_‘ + dateFor_file_name + ‘.json‘, ‘w’)

f.write (artistJSON)

f.close()

It works totally awesome — even for artists with unusual characters in their name such as Mötley Crüe. It doesn’t work so well for artists with exactly the same name such as Alice Cooper and Alice Cooper. In the latter case, data for the second Alice Cooper overwrites the file and data for the first Alice Cooper.

I could use the ‘a’ (instead of the ‘w’) argument to append it but for it to merge the data how I want it* isn’t really worth the time and effort it would take to write that script. What I’ll do, instead, is write the PHP for putting both sets of data where it belongs in my MySQL database which I have to do anyway. What I noticed while writing that previous post was LastFM isn’t a pretentious wiener like the guys who work for Championship Vinyl in High Fidelity (and MusicBrainz contributors). LastFM uses only one of those MBIDs for  Alice Cooper. I just have to test both MBIDs again to see which one — then I can write … wait … I could add some … other string from an artist’s data to the filename … OR I could include the time in addition to the date … and that might be helpful for … something else as well. But, as I said, I need this PHP written as well.

Back to the primary topic:

I need a script (and maybe a … are they called “lookup” tables?) that knows to put data from both Alice Cooper MBID … crap … no, wait … because these are hard, quantifiable (yes, I know I’m mis-using that word … be thankful I didn’t say “mis-abusing”) numbers I can add them! That’s the whole purpose behind getting LastFM data in the first place!

I am hoping Joan Jett is equally easy. For some reason, MusicBrainz contributors have her under Joan Jett and Joan Jett and the Blackhearts. I have no idea why. She’s never done anything without the Blackhearts. This is not a Tom Petty and Tom Petty & the Heartbreakers situation … aw, man! … I also need to deal with Tom Petty.

Spotify has one artist id for all Alice Cooper albums as well as Joan Jett. They have different, separate ids for the two Tom Petty entities which I’m okay with (and I’m sure Spotify is relieved). There are other issues with Spotify both odd but whatever as well as inaccurate but … that’s a whole different, well, issue.

The Actual Tests

TEST #1 – Part 1

First, I’ll demonstrate the “problem” by showing the response to a request to MusicBrainz for the Alice Cooper group’s Release Groups (a list of albums containing all the different releases–like other countries, etc.–of that album).

ACgroup_MB_ReleaseGroups.png
The Alice Cooper band MBID data includes 25 release groups (for only 7 “real life” albums! Still only 8 if you include Greatest Hits).

OMG! I just noticed there is a “type” property with a “Group” value! Woo-hoo! That is going to make everything sooooo easy.

ACperson_MB_ReleaseGroups.png
MusicBrainz has 25 release groups for Alice Cooper as a solo artist. Exactly the same amount as the group which is odd considering he has 20 studio albums (which doesn’t include live albums or compilations). That … is weird.

I notice, with a smile, this MBID has a “type” of “person” key/value pair.

I have to note that the Alice Cooper discography Wikipedia page combines solo artist and group lists for studio albums, live albums, and compilations but separates group and solo artist singles. Alice Cooper is just a mess. 😉

Important part: Each MBID has 25 albums for a total of 50. Yes, I read all of that and know each MBID has a different list of albums.

TEST #1 – Part 2

Artist Info from LastFM using the person vs band MBIDs

ACcombined_LastFM_artistInfo.png

The LastFM response using the person MBID gets me artist info for the same MBID.

The LastFM response using the group MBID redirects me to the artist info for the person MBID and the data is the same — note the listeners and playcount values.

Moral: I only need to use one MBID for getting info from LastFM — the person MBID.

Let’s try Joan Jett.

JJperson_MB_ReleaseGroups.png
MusicBrainz has 9 release groups for Joan as an individual.

I’m amused they have hometowns (“begin_area”) for both Alice (Detroit) and Joan (Wynnewood) but for the Alice Cooper group (Pheonix) and JJ & the Blackhearts (Los Angeles) they have the city in which the band was formed.

JJperson_MB_ReleaseGroups.png
MusicBrainz has 25 release groups for Joan Jett and the Blackhearts!

Joan Jett & Company isn’t quite as simple with LastFM.

JJperson_LastFM_artistInfo.png
Joan Jett individual MBID stats from LastFM

Joan Jett the person is, apparently, similar to neither the artist who inspired everything about her (Suzi Quatro) nor her contemporary, Pat Benatar.  Okay.

JJband_LastFM_artistInfo.png
Joan Jett and the Blackhearts MBID stats from LastFM

JJ & the BHs are, unsurprisingly, similar to her old band The Runaways and, well, Lita Ford (lead guitarist for The Runaways).

Neither “solo” or with the BHs (and those should be, in truth, the same MBID) is similar to Evil Stig (a band for which JJ sang 100% the lead vocals, not just some as she did with the Runaways).

Most importantly and irritatingly, the MBIDs have different values for listeners and playcount. I can still simply add them but this requires me to fetch each MBID in Last FM separately (for the time being, while I’m using my Python script to put the responses into JSON files).

Update: I’ve made a little lookup-ish table.

queryForMBID.png

I know I don’t need the name columns, but it helps me. This is for artists that are identical in the real world to make sure data from LastFM goes where it should — Alice Cooper has all Alice Cooper stats, albums, and tracks and Joan Jett gets all her stuff.

Speaking of Joan Jett, I am about to–any second now–make a lookup-ish table for related artists so I don’t need my “manual” group charts below.

steveTaylorGroup.png
This chart uses arrays containing an artist and groups for which they were the only lead singer.

On a related note, I’ll make a “related albums” table for artists who were only on some of a band’s albums — for example, any Dio-related query knows to get only those Rainbow and/or Black Sabbath albums for which he was the lead singer.

An issue that will still remain is albums for which an artist is only on one or some of the songs:

  • My albums table won’t accept albums if the artistID exists in the artist table which means neither The Rocky Horror Show nor The Rocky Horror Picture Show get inserted because they’re cast albums with various singers–Meat Loaf‘s artistID won’t match the artistID for those albums and “Various” or whatever isn’t in the artists table.
  • Ted Nugent‘s Free for All album doesn’t display for Meat Loaf queries despite Loaf singing some of the songs — he isn’t the “artist” for the album. Hmm … I should look and see if he shows up in any of properties for that album in any of my data sources … my goodness but this app is becoming wicked cool wicked fast!

Footnotes

*Once I increase my MongoDB and CouchDB knowledge, perhaps it won’t be such a time-consuming task.

Using the Last.fm and MusicBrainz APIs

I am quite pleased with my app that uses the Spotify Web API for creating data visualizations using the popularity data of artists, albums, and tracks. That popularity score, however, is a relative number so I can’t truly compare much in a useful way, however pretty my UX is.

Now is the time on Sprockets where we start a new project using the Last.fm API to get real, quantifiable data — the number of listeners for each entity (artist, album, track) and playcount for each. I’m very interested to find how those numbers differ.

Spotify and the Spotify Web API make it very easy to get ID numbers for each artist, album IDs for each artist and track IDs for each album. In fact, the documentation shows they might even be making an effort to be stupid. The example URL in their documentation for getting info about an artist is:

/2.0/?method=artist.getinfo&artist=Cher&api_key=YOUR_API_KEY&format=json

So any spelling errors or similar names will make your life miserable. Fortunately, their database uses (for the most part–I’ll get to that in a moment) the mbid or MusicBrainz ID for everything so you can, instead, write the above like this:

/2.0/?method=artist.getinfo&mbid=bfcc6d75-a6a5-4bc6-8282-47aec8531818&api_key=YOUR_API_KEY&format=json

Unfortunately, there’s no Last.fm equivalent to Spotify‘s utterly delightful endpoint for getting all albums by an artist:

/artists/{id}/albums

Fortunately, the MusicBrainz API has such an animal but it’s a little more complicated and it took me a bit of trial and error to figure out quite how it works. I’d like to share it with you. Here are the steps I’ll need to complete to get all of the information about an artist, their albums, and the tracks from those albums with an extra curveball just to make it fun.

Step #1 – Search MusicBrainz.org for the desired artist

In this case, I chose Alice Cooper.

aliceCooperSearchResults.png

For the uninititated, not only would using the search term “alice cooper” bring up tribute bands but the fans (I count myself among them) who insist on differentiating between the group and the solo artist mean I have to use not one but two MBID numbers for this single artist.

Step #2 – Get the Artist’s MBID from MusicBrainz

Click the artist name in the search results then click the Details tab on the artist page.

aliceCooperDetailsTab.png

Step #3 – Get the Artist’s release-groups from MusicBrainz

Copy the artist MBID and paste it into a MusicBrainz web service URL using the artist entity and a ?inc=release-groups parameter like so:

/ws/2/artist/ee58c59f-8e7f-4430-b8ca-236c4d3745ae?inc=release-groups&fmt=json

to get a list of “release groups.”

AC_MB_info_browser.png

I broke up with Dreamweaver a while back, dumping it for the far smarter, cuter, cooler VS Code but ol’ DW will always have a special place in my heart because of cool commands like Edit > Code > Apply Source Formatting which does this:

beforeAfter.png

Remember, you have to do that twice for Alice Cooper. Because we, his fans, are pricks like that.

Step #4 – Get the Artist’s Releases from MusicBrainz

A release is an album — like Zipper Catches Skin or Billion Dollar Babies. A release group is all editions and versions of that album. To see exactly what that means, we copy the id for the Constrictor release-group from line 26 in the above image and construct the URL below using the release-group entity and inc=releases parameter.

/ws/2/release-group/24c4fdde-4494-3837-a288-5b4fdbe966eb?inc=releases&fmt=json

That gets us all six releases in the Constrictor release-group:

  • United States with “packaging”: “Cardboard/Paper Sleeve” (a vinyl record)
  • United States with “packaging”: “Jewel Case” (a CD)
  • Japan CD
  • United States with “packaging”: “” (digital?)
  • Germany with “packaging”: null
  • United Kingdom with “packaging”: null

I don’t know what null packaging means.

The first time I get data from Last.fm, I’ll try all of these and hypothesize “Cardboard/Paper Sleeve” will have a zero playcount.

Step #5 – Get a Release’s Recordings from MusicBrainz

A recording is a “track.”

Construct a URL using the individual release entity and inc=recordings parameter.

/ws/2/release/1189de93-7e9f-3b47-a9df-e2599e62f75d?inc=recordings&fmt=json

The UK release of Constrictor has 10 tracks. There will probably never be a deluxe or remastered edition. Which is a damn shame.

Step #6 – Get an Artist’s data from Last.fm

Using the artist MBID, method=artist.getinfo, and an API Key as parameters, assemble a Last.fm URL like so:

/2.0/?method=artist.getinfo&mbid=ee58c59f-8e7f-4430-b8ca-236c4d3745ae&api_key=YOURAPIKEY&format=json

The resulting JSON for Alice Cooper the man includes a bio, birthday, similar artists, genres, and–for me–the important bit:

"stats": {
"listeners": "1362113",
"playcount": "31011489"
}

You’re dying to know how that compares to Alice Cooper the group, amiright?

"stats": {
"listeners": "1362113",
"playcount": "31011489"
}

Well, check that shizzle out! We only need to run what will be this query once for Alice Cooper! Both MBIDs get the exact same data. Wanna know something nerdy and cool?

The artist entity URLs I used earlier to get the release groups differ …

  • Alice Cooper the person has a gender of “male.” Alice Cooper the band has a gender of “null.”
  • The person has a “begin_area” of Detroit, MI while the band has Pheonix, AZ
  • The band has a life-span of 1964-1975 with “ended”: true while the person’s life-span begins with “1948-02-04” and ends with “null” and “ended” is false.
  • Type properties are “group” and “person”

Step #7 – Get Album Info from Last.fm

We’ll go with Constrictor again. Using album.getinfo this time.

/2.0/?method=album.getinfo&mbid=1189de93-7e9f-3b47-a9df-e2599e62f75d&api_key=YOURAPIKEY&format=json

  • listeners: 64,771
  • playcount: 545,070

For comparison (the entire purpose of this whole exercise), let’s also get Billion Dollar Babies:

Well, I’ll be horn-swaggled! There were three Billion Dollar Babies release-groups in my JSON file but none of their IDs worked so I searched at MusicBrainz and got the actual mbid for the release-group which had 13 releases in it! How do you pick? Eventually, I’ll get all of them but …

First choice didn’t work … this is going to get tedious …

Okay, second try gets me some stats

  • listeners: 169,768
  • playcount: 1,222,112

Yeah, see that’s a significant difference.

Step #8 – Get Track Info from Last.fm

I went for “The World Needs Guts” from Constrictor using the track.getInfo method as a parameter.

/2.0/?method=track.getInfo&mbid=70bd458f-c62d-4014-99e7-233c3c0b8f96&api_key=YOURAPIKEY&format=json

  • listeners: 11,332
  • playcount: 38,656

Which I’m sure is tiny crumbs compared to even one instance of “School’s Out” let alone adding all of the recordings of “School’s Out” together.

Conclusion

It will take me a lot longer to get the data from Last.fm that I need than it does to get data from Spotify but it will also be data I can use!

Prez Play Pro Progress

Thus far, if I spend this much time fixing stuff, it’s really frustrating because I’m not learning anything, I’m just trying to figure out what’s broken and why — but it wasn’t my own mistake or lack of knowledge.

This time, however, it is TOTALLY my lack of knowledge so each time I hid a bump, I learn something. Each problem I solve — I learn how and why what I did wasn’t working and how to do it correctly!

A lot of my work super-recently has been learning and using Javascript‘s map, reduce, and filter methods. I was depending on SQL queries (my database being MySQL, if you’re curious … though I was using MariaDB before changing hosting plans) to get the exact results and data visualization I wanted but, for multiple reasons, I’d prefer getting all the results from my database and manipulate the data client-side with Javascript.

Much of what follows is actually from an email I just sent to the American Nazi Party replying to an answer they sent to a question I asked a couple weeks ago. Say what you want about nazis, whenever I’ve asked a question, they’ve sent a polite and meaningful answer. Not all candidates/parties are responsive, let alone courteous. Anyway, I was explaining why I asked my questions and what I was doing.

So, here are my latest screenshots along with what is still in progress, etc.

If only what I consider “right” candidates run — excluding Trump because, of course, he’d win every state, then Gary Johnson wins every state. If we exclude him as well, Darrell Castle (usually Constitution Party) wins all states in which he runs except Arkansas where Jim Hedges wins. The gray states have no conservative/right candidates.

Screen Shot 2018-11-17 at 10.17.24 AM.png

Whatever happened to the Natural Law Party and Reform Party? Maybe the former are all write-in candidates (Darn it! I need to get write-in data!). I know the Reform Party was only on a couple state ballots in the 2012 election — not only that but with two different candidates. I read somewhere at least one state Reform Party nominated a different candidate because, despite how conservative Andre Barnett was (and still is, I assume), they didn’t like that he was (and still is, I suppose) black.

If only “left” or even just socialist candidates run, Jill Stein wins every state except Nevada, South Dakota, and Oklahoma (off the top of my head I think it’s simply because no “left” candidates besides Clinton were on the ballot there). If we only include socialists except Jill Stein, these are the results:

Screen Shot 2018-11-17 at 10.24.07 AM.png

Michael Maturen looks like he wins the most states (I don’t have electoral college votes in the code yet) and there are five tied in second place for number of states. Again, gray means there are either no socialist candidates in those states (except Stein) or, if there are — I need to clean up my data to make sure — there are candidates but none received any votes (which I find difficult to believe).

Once I get these all working, I’ll start adding previous elections and, hopefully, the next one won’t take two years to put together (in my own defense, most of that two years was the learning curve).

Refactoring Artist Management in MyRockinApp Part 1

Adding artists has been an ongoing pain in my ass during the entire lifetime of this project for many reasons. All of those reasons have one root reason — when I want to get something done, I tell myself I don’t have time to learn how to do it better … and I spent all that time learning how to do it the way it is now … I want to keep practicing that …

TL;DR — I am taking the time to completely refactor (as well as redesign) the entire app so it is more efficient, elegant, and easier to use.

The horrifying current state of affairs

I add artistIDs to a new array. Usually, they’re artists I have yet to add, artists I’ve realized I didn’t have, or there’s a new class of Rock and Roll Hall of Fame nominees.

nominees.png

Then use the Spotify WebAPI to get their data and insert rows into the artists table of my database. I do this the most stupid way possible — by changing the value of a variable in a PHP file and then going to that URL in a browser.

Then, I … don’t judge me! … I’ve been manually adding lines like this …

artistInstances.png
Yes, I know my camelCase wasn’t consistent. That’s why I’m doing and writing this.

So that … um … I can … well … manually add them to the “Choose Artist” menu like this:

artistMenu.png

Needless to say, I’ve often remembered to update one of those files and not the other. Or mismatch artistIDs and artistNames.

Because my albums table has a foreign key for artistID (from the artists table) and the tracks table has a foreign key for albumID from the albums table … sometimes forgetting to add things in order of artist, albums, tracks is another pain in the ass.

Also, I have separate files and functions for each little task and purpose — and not in the good way you’re probably picturing in your mind. They’re all redundant and overlap in countless ways.

I’ve know there was a better way for a long time but I’m always busy and always in a hurry. But I’m fixing that now. I’m forcing myself to. There’s a growing list of things like adding genres and grouping artists i.e. Dio, Ronnie James Dio, Elf, Rainbow, Black Sabbath, etc. that — to keep my sanity intact — require me to create some forms and functions resembling code written by a professional.

I’m doing this now for two reasons — I have the time and patience … but, more importantly, I learned a mother-butt-load earning Udacity’s Mobile Web Specialist nanodegree as well as the other resources I discovered while completing the program.

Now is the time on Sprockets where I replace much of the PHP in the app with Javascript. All of the Front End stuff, in fact, will now be strictly HTML, CSS, and Javascript. The PHP will be hidden in the back like Geoff Nicholls and any other heavy metal keyboardist.

RoxorRescue for Disaster Search & Rescue

Graduating from the Mobile Web Specialist nanodegree program gave me the confidence to enter my first hackathon. The IBM Call for Code hackathon started at least two months before I heard about it so I’m only going to get a few evenings and weekend chunks of time in the now two weeks before the deadline but … I’m loving the project I’m working on.

Idea #1 would have required months of research and whatnot but it’s possible I could finish this (to some extent) in time to submit it.

RoxorRescue is an offline-first progressive web app using PouchDB and CouchDB. Submissions have to use IBM Cloud tools so I’ll be using Cloudant (for the CouchDB bit) among other products.

Here’s a concept for the GUI I made in Photoshop along with the current state of affairs.

Woo.png

It’s meant to be used on a mobile device and when I tested it there I was elated that the photo bit finally worked but …

notWoo.jpg

… a couple things stuck out … not the least of which was the inaccessible “Take Photo” button.

Post-hackathon, I’ll be welcoming any and all contributors during Hacktoberfest and beyond.

I Can Feel the Power

Geeky Iron Maiden data is at the bottom.

Learning so much in the Mobile Web Specialist nanodegree. My Dev Tools skills have exploded. For example, I’ve struggled for hours (all together over the course of many days) to figure out what’s making my pretty new album chart display with either way too much margin:

idol
Note also the horizontal scroll showing how much space is there for nothing.

or, when I tried (among other “fixes”) the fix that did anything remotely desirable, up against the left with no margin:

billyIdol

Well, today, I used my little web app to look something (Iron Maiden related — see below if you’re interested) up and — a combination of irritation and actually having the time — I fought with it until I fixed it.

The chart is in a DIV with

id="forChart"

No matter what style in CSS or property in D3 I changed, nothing seemed to work. I also tried removing various DIVs and, opened my dev tools and saw that the page was still seeing a style for #forChart I’d commented out a while back. Even though the page file itself was being updated when I refreshed, the CSS wasn’t!

Once that was fixed by clearing my cache, it was better but still didn’t look right so I uncommented the “container” DIV I’d commented out while debugging and that added some huge margins again. I was back to having Yucky Version A …

ironMaiden_2400.png

or Yucky Version B.

ironMaiden_wContainer.png

Just to make sure — because I was going insane — I at least identified the culprit was something in Bootstrap by removing all the Bootstrap:

ironMaiden_noBootstrap.png

I went back to Dev Tools and, while I couldn’t see anything affecting the margins, I scrolled all the way down to the little diagram showing the margins and padding. When you click the components inside, it highlights them in the browser so I could see there was a buttload of margins — I just had to figure out where. Then I realized that when I clicked in the code (the Elements tab of DevTools), it showed much more specific information under the Styles tab.

Then I finally got to do one of the coolest things Dev Tools can do which is hack around in the code to change what’s displayed. So I could test it “live” without uploading, refreshing, testing, changing the code again, etc. … OH MY GOD what a time saver!

Once it was beautiful, I copied the new CSS fromt the Styles tab, pasted it into my CSS file, uploaded, and viola! Awesomeness.

fixed.png

What made me visit my little app today was this … the sort of nerdy curiosity about data that inspired the app in the first place.

Geeky (Geekier) Iron Maiden Data

I was celebrating yesterday’s purchase of Iron Maiden‘s Live After Death by pricing some of the other albums on Amazon — in particular, Brave New World because it is my second favorite album of theirs and it has a wicked cool cover similar to Live After Death.

And this nonsense is what I saw …

nonsense.png
Can you see it? What a … a travesty! an injustice!

“Blood Brothers” — obviously the best song on the album, has like zero popularity. I couldn’t even hum what Amazon says are the most popular tracks from memory. I almost lost my faith in humanity. So, I went to my little app to see how the song rated on Spotify.

First I sorted all their tracks by Track Name.

myspotBlood.png

Various versions of “Blood Brothers” do seem to be a bit low on the curve. For comparison, I sorted by popularity. Here are their top tracks as of last Sunday.

topMaidenTracks.png

Then I sorted by album so I could see how the song placed compared to other tracks on the album — the way Amazon showed them.

braveNewWorld.png

On Spotify, it is the second most popular track on the album and by a decent margin.