Creating a data visualization tool using D3.js, Crossfilter and Leaflet

I’ve recently completed my first javascript data visualization project at The Sentinel Project for Genocide Prevention to develop a dashboard that tracks indicators of hate crime in Iran.

You can take a look here:

Screenshot of Threatwiki Data Visualization

Initial requirement

I had a set of data coming from a JSON REST API and I wanted to show the main data in a table, display the geographical coordinates of the data on a map and be able to filter by date and tags.

Technology choices

I chose Crossfilter to be able to filter through the data, D3.js to generate/display all the data and the map itself is created with leaflet.js

Implementation details

For the map generated with leaflet.js, I used CloudMade (which I highly recommend) to get maps from OpenStreetMaps. I added a layer on top of the map to show different regions of the country, which was coming from Natural Earth shapefiles and converted to data with TopoJson and added on the map with D3. Leaflet turned out to be a great tool that I will definitely use again, I used the D3 + Leaflet tutorial from Mike Bostock to learn how to combine the 2 tools together. To add the datapoints at specific locations on the map, I used the marker functionality of Leaflet. I started adding the datapoints manually on the map by generating the proper svg tags with D3 on the canvas but I switched to markers because this will allow me later to integrate with LeafLet Market Cluster. This Leaflet plugin gives you the possibility of combining multiple closely located points on a map, which becomes useful when you have a high concentration of points in a a small region that are hard to distinguish at the normal zoom level. This other tutorial Let’s make a map by Mike Bostock  also come handy when working with D3 and maps.

There was a good learning curve necessary to understand D3.js. Reading code from examples is not always enough, I had to read about the core concepts and the introduction available on their github wiki. However, once you understand the concept, you realize that this is a very powerful tool to generate all kind of visualization either visually (in a canvas using SVG) or just displaying a set of data as text.

To be able to filter by tags, date, and other metadata, I used Crossfilter, a open-source library developed by Square. To learn about Crossfilter, I used this excellent example on their website. I used their example code to create the visual bar chart that is used as a timeline. There is also a great tutorial on the Wealthfront Engineering blog that really helped me to understand the concepts of the library.  Crossfilter was definitely easier to grasp and understand. The most recent version of the tool, 1.2.0., released just a few days ago, is definitely necessary. They introduced the concept of filterFunction, to give you more control on how you filter your data.

For a typical filtering in Crossfilter, you only provide the object to the filter function of a chosen dimension and Crossfilter will do the filtering. You  need to implement a filter function when your data structure gets more complex. Let’s say you have a record that can have one or multiple tags. If you only filter by giving the name of a tag, you will only get back the records that contain very specifically only this 1 tag and ignore all the records that have this tag combined with other tags.

So you want to implement a filterFunction like this:

window.filter = function(tagname) {
 byTags.filterFunction(function (tag) {
  if (tag!==null && typeof(tag)!='undefined'){
   for(i=0; i<tag.length; i++) {
    if (tag[i].title==tagname){
     return true;
  return false;

In that case we will go through our array of tag inside the record and return true to the filter at the moment we find the matching tag inside the array, so it wouldn’t matter if we have 1 tag or 10 tags on that record.

To learn more and get involved

If you are interested to look at the code, our project is hosted on Github. You can also look directly at the visualization javascript file. The data for this project is free of access to anyone who wants it, drop me an email if you need information on that.

That visualization for The Sentinel Project for Genocide Prevention has been announced on their blog.

If you have some experience with data visualization, maps, GIS or you just are a data nerd send me an email, we have lots of interesting projects coming up!


7 Responses to “Creating a data visualization tool using D3.js, Crossfilter and Leaflet”

  1. yo Says:

    What a nice dataViz project

    A small remark : scale are not given throught D3 histogram and that should be nice.
    Good choice for, techno and awesome work ! Well done.

  2. jeromegagnonvoyer Says:

    Thank you! Any reason to use D3 histogram instead of Crossfilter? Here on this thread!topic/d3-js/P73nU8r3GAU the creator of D3 mentioned there was no reason to use D3 histogram when using Crossfilter. Just curious what are your thoughts

  3. Adam Pearce Says:

    Thanks for writing this up! The idea of a crossfilter + map has been kicking around in the back of my head for a while and seeing your post inspired me to make this:

  4. New Visualization and Hate Crime Mapping Tool Can Help Prevent Genocide Says:

    […] Interested in learning more? For technical details of this data visualization project, take a look at Jerome’s blog for information on using D3.js, Leaflet.js and Crossfilter. […]

  5. david.d3 Says:

    good work, thanks.
    it will be good to have the same for other countries and then have a comparison and a table of which country is the top!

  6. Asheesh Says:

    This is really useful information. I’m trying to do something similar albeit for presenting zoomable US maps. I had a question regarding point about using maps from cloudmade. If I were to present a similar presentation as yours on US states, is the topojson readily available that I can use with leaflet? If not I’d appreciate if you can provide pointers on the way to go about it.

  7. Robert James Liguori Says:

    I’ve listed your crossfilter video here, in my list of crossfilter tutorials… thank you for producing the nice article for the crossfilter community.:)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s