depthq - ccm|mac

I have always been interested with visualisation of data. I like to generate graphs from data. I have some ideas in my head for a while but never got around to actually doing it. Google Finance has a really nice visualisation for tracking stock prices and foreign exchange. So since I am interested in tracking the foreign exchange of the local banks in Malaysia I decided to do a bit of hacking to make my own Google Finance-ish interface to the data. The graph that I render is actually the percentage of change as compared to the forex rate of 4.77.

I've been scraping the foreign exchange data from RHB Bank website for a while and wanted to plot the data into a graph. I looked around a bit for some open source graphing and data visualisation tools and found Flot to be quite nice and its developed by some Django and jQuery fans, so maybe I am biased. Flot is a pure Javascript plotting library.

So I have a cron script that will scrape data from the RHB Bank website and writes it into a CSV file. I used BeautifulSoup to easily extract the data out of the HTML. I setup another cron script that will use the data from the CSV file and it will generate the HTML that will contain all data that is needed by Flot to draw the graph.

Flot is pretty easy to use and the basic conventions are pretty good. Since its a pure Javascript library all the code is on the HTML page and you can check it out if you want. I was thinking of adding it to Github gist but I think its not necessary at the moment since the code is very specific to my own needs. The documentation and examples are pretty good and its quite easy to learn. I particularly like the autoscaleMargin property for configuring how much margin it should automatically add to the axes so that the min and max value for the data would not be rendered on the edge of the grid. I also particularly like how easy it is to do selection and zooming. Just a few lines of code and its done.

So why did I mention I used a bit of Django?

I was lazy to find (and learn and setup) another templating language and decided to use Django's templating engine instead since the setup of that server that I am using works with Django. The HTML file that is generated is rendered off a Django template file using render_to_string() and I just pipe the stdout to a file. The only unsavoury thing is that I need to setup the DJANGO_SETTINGS_MODULE=settings and the settings.py file has only the TEMPLATE_DIRS setting.

Okay, so the next step is to add some news feed with the forex data to put more context while looking at the data. I guess that would be another weekend and another blog post then.

I have been looking for a more elegant way to code and structure a Django view so that it will support rendering of responses encoded in either HTML or JSON.

I do this because I want to add some AJAXy usability to my HTML forms while retaining some backwards compatibility for non-Javascript enabled Web browsers.

First, lets look at a simple Django view:

def foo(request):
    if request.method == 'POST':
        form = Form1(request.POST)
        if form.is_valid():
            form.save()
            return HttpResponseRedirect('/foo/success/')
    else:
        form = Form1()
    return render_to_response('app1/template1.html', {'form': form})

I was lazy and did some hacky stuff like this to support JSON response.

def foo(request):
    if request.method == 'POST':
        form = Form1(request.POST)
        if form.is_valid():
            form.save()
            if request.POST.get('type', '') == 'json':
                data = {} # define your own data structure
                return HttpResponse(simplejson.dumps(data),
                                    mimetype='application/json')
            else:
                return HttpResponseRedirect('/foo/success/')
    else:
        form = Form1()
    return render_to_response('app1/template1.html', {'form': form})

And for the HTML form, I needed some Javascript (using jQuery) like this:

    $("#id_cmd_submit").click(function(){
        $.post("/foo/", {type: "json"},
            function(data) {
                alert(data);
            }, "json");
    });

I was thinking to myself that this type parameter is kinda of ugly and not necessary. The HTTP protocol actually has a Accept header that you can use to define what encoding the HTTP response from the server can be and this is automatically added when you use the $.post() from jQuery using the json type.

If you examine the HTTP headers in the Django view through request.META['HTTP_ACCEPT'] you will see values like this:

application/json, text/javascript, */*

So to make the hacky method a bit less hacky, I did this:

def foo(request):
    if request.method == 'POST':
        form = Form1(request.POST)
        if form.is_valid():
            form.save()
            if 'application/json' in request.META.get('HTTP_ACCEPT', '')
                data = {} # define your own data structure
                return HttpResponse(simplejson.dumps(data),
                                    mimetype='application/json')
            else:
                return HttpResponseRedirect('/foo/success/')
    else:
        form = Form1()
    return render_to_response('app1/template1.html', {'form': form})

And the Javascript now looks like this:

    $("#id_cmd_submit").click(function(){
        $.post("/foo/", {},
            function(data) {
                alert(data);
            }, "json");
    });

Of course the code is still not very pretty and there are better ways to structure it better. In fact, all I did was just to make the API more "conforming" to HTTP and thus the Javascript code slightly cleaner. The Python code is somewhat the same.

As for the Python code, I see that someone has tried to make this style a bit prettier but I am not too sure if it will fit with all my use cases and I don't like the additional complexity.

I did read a bit about Adrian Holovaty's take on this AJAXy stuff. Its definitely cleaner than my approach but I wish Django have better support for HTTP's Accept header since I like the approach of using the Accept header. I found a middleware in Django snippets that can abstract away the complexity a bit in a more elegant way.

There is a good writeup on REST worst practices by Jacob Kaplan-Moss which touches a bit on using the Accept header instead. Malcolum Tredinnick also though on this a bit in an article about content types on this blog.

So I guess I did not bark up the wrong tree. And surprisingly, those blog articles that I found were quite recent. At least I am not outdated.

Almost 2 weeks ago I was in Vilnius to attend EuroPython 2008. It was my first time in Vilnius and first time to attend a conference about Python, which is the main programming language that I've been using for the past 2 years of work at Exoweb.

It was interesting to meet so many people that are also using Python everyday in their work and projects. In the conference, I met up with Greg and Tomasz, who's Greg's partner in crime in the software business. They are part of Enpoka that is doing outsource software development. I also met up with Tobias, who drove all the way from Tromsø to Vilnius so that he can fix his car in Riga, its more than 2000+ kilometres, pretty crazy!

I attended some talks on Jython and PyPy. I have to say I am pretty impressed with how far Jython has come (and only recently realised the state of development of the JVM in the dynamic languages world) and the grandiose plans for PyPy.

The day ended with Guido's keynote but it was done through video conferencing and I guess it wasn't to the best effect. It was regarding Python 3000 which I've heard at least twice so far.

The next day started with a talk on a tool to visualise relationships between Python objects and duplicate code finder a.k.a CloneDigger. I listened to a talk on LEAN software development by Steve Alexander of Canonical but it was a bit too vague to me, although I thought his approach of using photographed stickies as slides was cool. There were some good talks of Python's more advanced topics (descriptors) and the internals of some of the standard data structures we use everyday like lists, dicts and sets.

I attended the Cython talk but I was getting distracted hacking away with Greg on using greenlets with twisted. Both of us were not so happy with the deferred style of coding asynchronous servers in twisted and wanted to find a more manageable approach. We have a pretty simple prototype and I would want to apply the same idea to a larger code base .

The day ended with a keynote by Hans Rosling, who I admire quite a bit due to his great TED talk he gave. He is more impressive live and I truly enjoyed his talk although it wasn't much about Python. There were some lightning talks before the keynote and the one that I can remember very clearly is gSym, which is a python editor that visualises your code in different ways such as mathematical formulas and lisp style.

For the last day, one of the highlights was a talk about hacking your dna with the amazon cloud. I talked a bit with Mike the DNA hacker and learned quite a bit about his travels in South East Asia which were pretty cool. After the hacking on greenlets and twisted, Greg decided to hold an unconference session on it to discuss about it. Tommi, who knows quite a bit about twisted gave some good insights on the greenlet+twisted approach which is that although it works its dangerous.

There is a lot of material that I would like to catch up on but I haven't found the time to really dig into it. I really enjoyed the conference and thought it was a good chance to learn a few things and meet similar minded people.

I managed to meet briefly with some Django contributors and Jim Baker who's working hard pushing out Jython 2.5. He says he'll be in China in September, maybe he'll come over and give a talk on Python and Jython. I am kinda interested in Jython myself and have been submitting some bug reports, hopefully its useful for them . I was not really ready for the Django sprint and spent a bit of time looking into Django new forms admin. At least I learnt a bit about the new branch that will go into 1.0 soon.

I have bookmarked some of the more interesting stuff I learned from the conference in del.icio.us and also took some pictures while sightseeing.

I have more to say, ok maybe not. Over!

Leave a comment

I've been trying to setup Django on my shared hosting and haven't had any success until today. Finally, I managed to get it running and you can check it out here. Its very simple at the moment. Will host the updates for the ExoBlog development on its own blog, got to eat your own dog food!

I wrote not long ago about running Django 0.95 on Windows. I've since stumbled on a few more issues with Django:

1. Timezone issues
   
   No matter what timezone you set in settings.py, it won't work properly because there is a bug. Get the patch from ticket 2315 to fix the problem. If you are using sqlite3, the timezone that is used for Beijing time is 'Asia/Shanghai PRC' on Microsoft Windows (please refer to UPDATE2 below). On the other hand, if you are using postgresql, you need to change it to 'Asia/Shanghai'.
   
2. Support for Unicode filenames
   
   The default behaviour for uploading files with Unicode filenames is to use the following regex to replace characters before saving it: [^-A-Za-z0-9_.]. This might not be the behaviour that you might want. There is some discussion here in ticket 1355. It seems that the filenames in Windows NTFS filesystem support Unicode (utf-16). I've made some changes to the Django source to solve this problem. I am sure I haven't covered all the cases but it works for what I want it to do for now.

Another issue that I've encountered when running on a Apache + mod_python + Django + PostgreSQL setup on Windows is that you might run out of ports.

WHAT? How can that happen?

Basically, what happens is that Windows default TCP/IP subsystem don't release the ports until its idle for 4 minutes and there is a limited range of ports that can be used by user programs. So, if you are making a large number of requests to the web server for a short period of time, Django might fail to open new connections to the database. It can be changed by tweaking some registry settings.

Check it out here for a detailed explanation and discussion about this issue. I am not sure, but maybe this problem can be solved with some caching for database server with high traffic. Hmmmm, running a Windows server on a high traffic site with default configs? I don't think so .

UPDATE1: I messed up the previous post and this is a repost. Oh bugger!
UPDATE2: Seems like the 'Asia/Shanghai PRC' timezone for sqlite3 doesn't apply to my Debian GNU/Linux setup at work.