Little Data

---

Data Visualization on the fly

@phil_renaud

Slides:

bdc.riot.industries

1. Why do we care about data visualization?

We're pretty good at recording data.

Too good, probably.

• On an average day, Twitter generates about 500 million tweets. That's about 15 times the number of books in the Library of Congress
• In August 2010, Eric Schmidt, former CEO of Google, said that between the beginning of time and 2003, humanity generated about 5 exabytes (5 x 10¹⁸ bytes) of data.
• Now, we generate that much data roughly every 2 days.

These mountains of data add up. We record much more data than we can possibly analyze 1:1 in a lifetime.

Even when we aggregate data, it's not always clear what's going on.

And this is sort of a bummer, because

Data is only valuable when it's understood.

Anscombe's Quartet

Anscombe's Quartet

Statistical Detection of ... ?

source

Statistical Detection of Election Fraud

source

Visualization lends meaning to raw data.

2. Data Journalism

• In addition to recording data at an unprecedented rate, we've become very good at exposing it as well.
• City open-data policies, like Halifax Open Data
• Scientific data, like The 1000 Genomes project
• APIs as interfaces to otherwise obfuscated data, like Census Reporter
• Github repositories of data behind investigative news, like FiveThirtyEight's Data

• This access provides us with the means to think critically about the world around us.
• If you know even basic spreadsheet use, you can start using the aforementioned data to research the subject of your choice.

• Journalists have taken this to its logical next-step: Data-Driven investigative journalism.

• Sites like FiveThirtyEight and NYTimes' The Upshot give us ways to think about their data.

• In the absence of the ability to code and analyze the information ourselves, Data-Driven Journalism sites do us a great service when they provide methods for us to interact with their data.
• A mainstay of Data Journalism is the pairing of a story with the understanding that a reader's interpretation of it may differ.
• In practical terms, this means that the best data journalism allows the reader to sort, filter, combine and rearrange data through a visual interface.

3. Data as a means of Critical Thinking

Some people mislead in order to persuade.

This probably doesn't work anymore. (good)

Persuade the right way: back your narrative up with data.

Telling Data Stories is more about being able to answer questions than about telling a single narrative.

(Data Science is not an Op-Ed piece)

There are some very good reasons to use data / data visualization as an aid to critical thinking:

1. Some of our questions have conflicting answers.

• It is a foregone conclusion that someone selling you something has an unobjective bias toward their product.
• 

  In the name of persuasion, some sources are simply unreliable. Sometimes they look like the wrinkle ad from before. Other times they require analysis.

• Data lets us explore, lets us fact-check.

Data makes critical thinkers of us.

2. Sometimes we ask the wrong questions.

• We might ask "How many customers did you have in October?"
• We probably means "How many customers did you have in October, relative to September" or "relative to last October"

Data answers the questions we didn't know we had.

3. Sometimes the answer to a simple question is unsatisfying.

• Q: How many Californians relocated from outside the USA?
• A: About 28%
• Okay but...
• Q: Where do the other 72% come from?
• Q: What about Californians who leave? Where do they go?
• Q: How does this compare to last year? to 100 years ago?
• Q: How does this compare to the US as a whole? To any given state?
• nytimes.com/interactive/2014/08/13/upshot/where-people-in-each-state-were-born.html

4. Sometimes there are many distinct narratives to explore.

• nytimes.com/interactive/2014/10/27/us/is-the-affordable-care-act-working.html

Data makes critical thinkers of us all

4. Static Data Visualization:
Best Practices

• Certain types of visualizations are well-suited to particular data types:

• Bar charts are boring, but well-suited to allow us to quickly determine differences in quantity:

• Sometimes pie graphs are okay for this too, but be warned: at a glance, people don't see area, they see height.

• Treemaps show quantity, plus a sense of hierarchy

• Maps provide several ways to show physical location.

• Maps are particularly versatile because of our common understanding of their expected shape; differences from our expectations let us see a different kind of story:

• This cartogram shows expected popularion by 2300. It is only by changing our previous knowledge of the shape of the world that this has any effect.

• Time-Series charts show how quantitative values change over time:

• Scatter plots show us the differences between things in two dimensions

• Network graphs show us the relationships between things - for example, this map of all of the followers of @CollideHalifax

5. The Qualities of Highly Successful Interactive Data Visulizations

The State of the Union in Context

source

Bar Charts, Interactive

The Guardian - The state of our union is ... dumber

source

Scatter plot/ Time series, Small multiples. Allows user to parse and reveal.

NPR - When are people working?

source

Area Graph. Allows user to visually compare.

Musicians' Deaths by Age

source

Time Series. Allows user to filter based on category.

Bachelors Degrees by Demographics

source

Visualizing MBTA Data

source

Parallel Coordinates, Maps, Bar and Area. Allows user to explore relative and absolute time, and compare stations/trains.

NY Times: How the Recession Reshaped the Economy, in 255 Charts

source

Loose interaction; the act of scrolling shows comparisons between industries. Sparklines. At the bottom, Small Multiples show the fuller picture.

Bloomberg Visual Data: State-by-State

source

Map, Vertical Bar Chart, Time-Series. Allows user to dig deeper and cross-compare state data.

• The primary theme among the interactions presented here is that they allow users to complete a few common, basic actions:
• Sort: Representing the data in different orders can reveal trends otherwise unseen
• Filter: Only show the data that meets certain user-defined criteria
• Compare: Allow the user to take pieces of the whole and see how they relate.

• Why do these visuals feel so compelling, compared to traditional static media, or data in chart format?
• The general pathway to understanding:
  • Unstructured Information
  • Data
  • Structured Information
  • Knowledge
  • Wisdom
• Visualization helps bridge the gap between Knowledge and Wisdom. Interactivity prompts exploration, and exploration leads to understanding.
• And smarter users = good for everyone.

6. Some common pratfalls of Data Visualization

So, when I see a Visualization, can I be sure that the Data Story I'm being told is correct?

• Not always! Be critical.
• Mark Twain's 3 kinds of lies:

"Lies, Damned Lies, and Statistics".

Avoid these common deceptions when designing data stories:

1. Maps as population indicators

NYC has ~ the same population as the shaded area

Slate's Equal Population Mapper

2. Cumulative Time-Series Charts

3. Circles in general.

We're meant to see area; we commonly just see height.

4. Pie charts when dealing with close values

(Bar charts are boring but we're really really good at reading them)

5. Bar charts with arbitrary axis points

4. Seeing correlations when they're coincidental

source

7. Little Data

What's your preferred code editor?

• Sublime Text?
• Visual Studio?
• Eclipse?
• Vim?
• R Studio?
• What about Chrome?

A skill that data scientists often underdevelop is the ability to derive meaning from data on the fly.

• Sometimes you don't have your ideal tools available
• Even if you did, sometimes the person to whom you're telling a data story, doesn't understand your raw data.
• We're going to use one of the most maligned programming languages, Javascript, to build an on-the-fly data visualization toolkit
• Time to open some tabs!

• Head over to

  tinyurl.com/bdc-import

• Copy the import code

• When you have that copied, head over to

  tinyurl.com/bdc-trade-canada

• Open up your Chrome dev tools console - CMD + ALT + i
• (or right-click and select "Inspect Element", then hit ESC)
• Paste in your import code from tinyurl.com/bdc-import and hit enter
• Congratulations! You now have access to a very advanced data visualization toolkit.

Here's what you've just added to this page:

• jQuery: document traversal and manipulation. “What’s the parent element of the first table?"
• Underscore.js: Data manipulation and parsing. “What’s the total of the values in this array?"
• D3.js: Data-as-a-document and visualization. “Plot the min and max values as colors on the red-to-green scale"

Let's try manipulating our data table.

A sample excercise: Use jQuery to find a given cell and change its background color

						$('td').eq(150).css('background-color', 'yellow')
					

"Which year had the highest total imports?"

jQuery: Show me the all the elements in the last row:

						$('tr:last-child td')
					

lowVal = _.min(
  $('tr:last-child td'), function(obj,iter){
    return $(obj).text().replace(/,/g,'')
  }
)
		      

						$(lowVal).css('background-color','yellow')
		      

Let's build on this. Let's highlight the import trends of each product from year to year.

Let's iterate (map) over each row (product) and:

• 1. figure out the min and max values
• 2. use D3 to paint the min values red, max values green, and scale the in-between accordingly
• 3. Iterate (map) over each cell in the row and paint its background according to where it sits in the d3 domain

$('tr').map(function(iter,row){
  var max = _.max($(row).children('td').map(function(iter,cell){  return $(cell).text().replace(/,/g,'') }))
  var min = _.min($(row).children('td').map(function(iter,cell){  return $(cell).text().replace(/,/g,'') }))
  var color = d3.scale.linear()
    .domain([min, max])
    .range(["#c33", "#0fc"]);
  $(this).children('td').map(function(iter,cell){
    $(cell).css('background-color', color($(cell).text().replace(/,/g,'')))
  })
})
					

Pre-made snippet for you:

tinyurl.com/bdc-gist2

Other countries to experiment with:

http://tinyurl.com/bdc-trade-all

Some observations:

• 2009: obvious downturn across almost all industries
• Looking for a common exception? ctrl+f "Pharma" in almost every major country
• Canada/Mexico: Food up, household goods down.
• China: almost everything up
• Iraq: the year before the US unilaterally pulled out of the war, almost all industries had an uptick
• Haiti: noteable uptick in textiles (cotton, wool)
• Syria: downtick across the board
• Egypt: Fuel and Natural Gas dropped off the face of the earth
• Belize: Fruits are down, vegetables are up (Thanks, paleo diet!)

Other visualizations, other data types

Let's build a bar chart over columnar data

tinyurl.com/bdc-baseball

• Open up your Chrome dev tools console - CMD + ALT + i
• (or right-click and select "Inspect Element", then hit ESC)
• Paste in your import code from

  tinyurl.com/bdc-import

  and hit enter

Let's highlight the Batting Average column:

$('#teams_standard_batting td:nth-child(19)').css('background-color','yellow')
					

Now, use our iterative method from before to turn that column into a heatmap:

var column = $('#teams_standard_batting td:nth-child(19)')
max = _.max(column.map(function(){ return parseFloat($(this).text()) }))
min = _.min(column.map(function(){ return parseFloat($(this).text()) }))

  var color = d3.scale.linear()
    .domain([min, max])
    .range(["#c33", "#0fc"]);

column.map(function(iter,cell){
  return $(cell).css('background-color', color(parseFloat($(cell).text())))
})
					

Pre-made snippet for you:

tinyurl.com/bdc-gist3

Let's make a bar chart!

• Since we already know the "max" value, we can assign that a max-width bar.
• Every other value will be its ratio of that max bar's width

var column = $('#teams_standard_batting td:nth-child(13)')
max = _.max(column.map(function(){ return parseFloat($(this).text()) }))
min = _.min(column.map(function(){ return parseFloat($(this).text()) }))
barwidth = 250

column.map(function(iter,cell){
  $(cell).prepend('
');
  $(cell).children('.bar').css('width', parseFloat($(cell).text()) / max * barwidth )
  var color = d3.scale.linear()
    .domain([min, max])
    .range(["#c33", "#0fc"]);
  $(cell).children('.bar').css('background-color', color(parseFloat($(cell).text())))
})

$(column).css('text-align', 'left')
$('.bar').css('height', '10px').css('float', 'left')
					

Pre-made snippet for you:

tinyurl.com/bdc-gist4

Thanks!

@phil_renaud

Slides: bdc.riot.industries

Bonus links:

Functional Programming should be your #1 priority for 2015

Quick Guide to Making an Interactive D3 map

Data for the The President's Fiscal Year 2016 Budget

538's Open Data