Wednesday, April 3, 2013

Using MongoDb and Sleepy Mongoose and jQuery to save data for a D3 js dataviz

It took me a while to put all these pieces together, so in case anyone else is in the same position, here is what I did.  The code is a bit rough with very little validation...

I wanted to be able to save save different datasets for a d3 dataviz I was working on.  I could have done this in MySQL but as the data was in json already, I thought it a good opportunity to experiment with MongoDB and an HTTP interface.

Having installed from instructions here:

Saving Data

I first tried saving data I already had in javascript:

var request = $.ajax({
url: DB + "_insert",
type: "POST",
data: 'docs=[{"dataset" : "' + dataset + '", "links": '+JSON.stringify(dataitems)+'}]',
dataType: "json"

request.done(function(msg) {
});, textStatus) {
alert( "Request failed: " + textStatus );

I had a couple of issues here.  The first was trying to use jsonp to avoid cross domain problems.  I was getting this error:

            jQuery191001886391662992537_1364976848482({"ok" : 0, "errmsg" : "_insert must be a POST request"})

Jsonp converts the post to a get and Mongoose does not support jsonp, so you have to use dataType json.  There are a number of ways to avoid the problem and the comments on the installation page are helpful.  I chose  to start Mongoose using --xorigin parameter:

                         python --xorigin 

The second problem was formatting the data.  There needs to be a docs= or you get

            {"ok" : 0, "errmsg" : "missing docs"}

And it needs to be stringified or encodeURI to avoid a parse error.  In the above example, dataset is a string variable and links is an array.

So now my data was saving and I could check by doing:

Listing Data

So now I wanted to pull back a list of the datasets I had saved so the user could choose which dataset to use.  Ideally I want to only pull back the list of dataset fields, it's currently pulling back all the data for each record.

The html has:

<div id="datasets"></div> <button id="load">Load</button><

The javascript to populate:

var request = $.ajax({
url: DB + "_find",
type: "GET",
dataType: "json"

request.done(function(data) {
var results = data.results;
// build a select list from the data
var list ="#datasets").append("select")
.attr("id", "dataset");
.attr("value", function(d) {return d.dataset;})
.text(function(d) {
return d.dataset; });
});, textStatus) {
alert( "Request failed: " + textStatus );

Getting a specific Dataset

And finally, when the user has selected a dataset, retrieving the values:

$("#load").click(function() {

// load selected dataset

var request = $.ajax({
url: DB + "_find",
type: "GET",
data: encodeURI('criteria={"dataset" : "' + $("#dataset").val() + '"}'),
dataType: "json"

request.done(function(data) {
// needs some error checking here
links = data.results[0].links;
// redraw diagram here

});, textStatus) {
alert( "Request failed: " + textStatus );


The page where this was used:

Friday, October 5, 2012

A Cheesy Adventure

I made my first cheese.  It was creamy and tasty and so easy, so now I'm trying Mozzarella!  This was all thanks to the really excellent marketing of Lakeland.  If you don't already get their catalogues full or useful and thought provoking products (who has room for a special plastic dish just for soaking oven racks, and a special brush for cleaning your electric grill and panini maker! ) you can browse their shop here  So the catalogue with the picture of cheese on the front cover and the cheese kit on the first page caught my eye and I duly purchased the moulds and book.

I tried to find some non-homogenized goats milk but in the end used Glenisk goat's milk which worked just fine.  Here are the instructions:

1 litre Glenisk goat's milk - heat gently to 24 degrees (watch out, that doesn't take long) the remove from the heat and add 2 tubs of Glenisk goat's plain yoghurt (100ml total) the mix in 1/8tsp (a few drops) of rennet.

Leave to stand overnight and in the morning cut into 2cm blocks with a knife and pour into the foot of a pair of tights.  You won't find the tights in the usual recipe but I'd lost the muslin I'd bought and the tights worked just fine.  Save the liquid Whey that comes off.  The cats loved it or you can use it in the bread machine.  After a couple of hours mix the curds gently with a large teaspoon of sea salt and pour into the cheese moulds and put in the fridge to drain further.  By the evening the cheese was ready, about a quarter of a litre.  I scoffed it all within 24 hours!  It's quite bland so I'm going to try adding crushed garlic and chives to the next batch.

Now must go see how the Mozzarella is getting on!

Monday, July 30, 2012

Handy trick for using django runserver/testmaker on remote server

Thanks Bart for showing me how to have a local version of django running on my VPS.

On the VPS start django as required, eg

python runserver
python testmaker -a web

Now do this from a command line where is your vps ip:

ssh root@ -L 5984:localhost:8000 -N

enter the password and it should not come back with anything.

Now go to this URL and your website should start


Wednesday, July 25, 2012

What to do with a 17lb turnkey when there are only three of you for Christmas?

Due to unforeseen circumstances, we had less people and a bigger turnkey than planned. And we are not big eaters. So father butchered the monster into breasts, legs and carcass and we cut a small piece of one of the breasts and roasted that. Very tasty is was too and we split the leftovers between us to make at least a further mean each. So what was I going to do with the huge bowl of left over bits? After a bit of googling I came up with this recipe and am very pleased with it.

Turkey Burgers for the Freezer

Amounts approximate!

Cut up the raw turkey into chunks and whizz in the food processor to turn it into turkey mince.
20% (5 biggish onions) of turnkey mass in onions. Chop in food processor and fry gently and leave to cool.
10% (10 pieces) smoke streaky bacon add to onions and cook quickly. Didn't brown.
10% (1 large) cooking apple chopped.
1 clove of garlic, chopped finely.
1 inch cubed of ginger chopped finely.

Let the cooked ingrediants cool down and then mix all together.
Using polenta meal to coat the burgers, (Was going to use semolina, as recommended by Nigella for roast potatoes, but none in the shop. Polenta worked very well as an alternative.) I used a burger making device to form the burger and put on a baking tray coated with polenta. Freeze then take off the baking tray and put into plastic bags, ready for that need for a quick wholesome meal.

D3 Nest function - converting csv data to nested data for treemaps etc.

Just finished 8 examples of how to use the javascript visualisation library D3 nest function - notes to self really, but shared in case they are useful for others. to see the code to see all the examples run.

Thursday, July 12, 2012

Simple D3 examples to try

I learn by looking at examples and then trying to modify them to do what I want. I've started creating little gists to record how use different features of d3. In case they help others, here is a start. They can all be run in bl.ocks

Processing CSV strings and files and grouping with nest:

  • - processing csv string 
  • - processing csv file 
  • - nesting/grouping to two levels

Drawing Axes:

  • - simplest possible
  • - date axis
  • - prettified, angled date labels and uses extent
  • - rescale an axis using a transition

Monday, February 20, 2012

Getting to grips with D3 Visualisations

D3 is a great library for creating all kinds of data visualisations written by Mike Bostock, but until recently I just couldn't get to grips with it. But then I discovered Scott Murray's great series of tutorials that start with the basics and build up and Luke Francl's appropriately named D3 for Mere Mortals. I've done some basic charts and am now rewriting the Protembla project management visualisations in D3. A big thank you to Mike, Scott and Luke.