Wednesday, April 3, 2013

Using MongoDb and Sleepy Mongoose and jQuery to save data for a D3 js dataviz

It took me a while to put all these pieces together, so in case anyone else is in the same position, here is what I did.  The code is a bit rough with very little validation...


I wanted to be able to save save different datasets for a d3 dataviz I was working on.  I could have done this in MySQL but as the data was in json already, I thought it a good opportunity to experiment with MongoDB and an HTTP interface.

Having installed from instructions here: http://www.kchodorow.com/blog/2010/02/22/sleepy-mongoose-a-mongodb-rest-interface/

Saving Data

I first tried saving data I already had in javascript:



var request = $.ajax({
url: DB + "_insert",
type: "POST",
data: 'docs=[{"dataset" : "' + dataset + '", "links": '+JSON.stringify(dataitems)+'}]',
dataType: "json"
});

request.done(function(msg) {
alert("success");
});

request.fail(function(jqXHR, textStatus) {
alert( "Request failed: " + textStatus );
});


I had a couple of issues here.  The first was trying to use jsonp to avoid cross domain problems.  I was getting this error:

            jQuery191001886391662992537_1364976848482({"ok" : 0, "errmsg" : "_insert must be a POST request"})

Jsonp converts the post to a get and Mongoose does not support jsonp, so you have to use dataType json.  There are a number of ways to avoid the problem and the comments on the installation page are helpful.  I chose  to start Mongoose using --xorigin parameter:

                         python httpd.py --xorigin 

The second problem was formatting the data.  There needs to be a docs= or you get

            {"ok" : 0, "errmsg" : "missing docs"}

And it needs to be stringified or encodeURI to avoid a parse error.  In the above example, dataset is a string variable and links is an array.


So now my data was saving and I could check by doing:

         http://test.com:27080/sanky/dataset/_find



Listing Data


So now I wanted to pull back a list of the datasets I had saved so the user could choose which dataset to use.  Ideally I want to only pull back the list of dataset fields, it's currently pulling back all the data for each record.

The html has:

<div id="datasets"></div> <button id="load">Load</button><



The javascript to populate:


var request = $.ajax({
url: DB + "_find",
type: "GET",
dataType: "json"
});

request.done(function(data) {
var results = data.results;
// build a select list from the data
var list = d3.select("#datasets").append("select")
.attr("id", "dataset");
list.selectAll("option")
.data(results)
.enter()
.append("option")
.attr("value", function(d) {return d.dataset;})
.text(function(d) {
return d.dataset; });
});

request.fail(function(jqXHR, textStatus) {
alert( "Request failed: " + textStatus );
});

Getting a specific Dataset

And finally, when the user has selected a dataset, retrieving the values:



$("#load").click(function() {

// load selected dataset

var request = $.ajax({
url: DB + "_find",
type: "GET",
data: encodeURI('criteria={"dataset" : "' + $("#dataset").val() + '"}'),
dataType: "json"
});

request.done(function(data) {
// needs some error checking here
links = data.results[0].links;
// redraw diagram here

});

request.fail(function(jqXHR, textStatus) {
alert( "Request failed: " + textStatus );
});

});



The page where this was used: http://beautifuldata.eu/sankey/experiment.html