depthq - ccm|mac

If you want to check to ensure your SSL or non-SSL ports in PostgreSQL are working properly, read further to find out how.

For SSL support, you need to setup your PostgreSQL to work with SSL properly. Just turn on the ssl flag in postgresql.conf and then make sure you've setup the server.crt as mentioned in PostgreSQL docs about Secure TCP/IP Connections with SSL.

If you want force psql to use non-SSL for the connection, you need to set the PGSSLMODE environment variable. If you're using bash, do something like this:

export PGSSLMODE=disable

And then try to run the psql to connect to the server. I would suggest that you be explicit in your pg_hba.conf to specify that you want hostnossl, so that the server will not accept SSL connections.

If you got things configured correctly, it should work without problems. On the other hand, you can force it to use SSL, like this:

export PGSSLMODE=require

You should see a fatal error message from psql, stating that there is no pg_hba.conf entry for SSL on. You can use the same method to test for SSL-only connections by using hostssl instead in pg_hba.conf.

This environment variable works not only for psql but also for any library or tool that uses libpq. For example, I include the console output for testing the psycopg library. I have already set up the PostgreSQL to work with SSL only.

silviana:~ cheeming$ export PGSSLMODE=disable
silviana:~ cheeming$ python2.4
Python 2.4.5 (#1, Jul 15 2008, 23:37:00) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg.__version__
'1.1.21'
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
psycopg.OperationalError: FATAL:  no pg_hba.conf entry for host "127.0.0.1", 
    user "cheeming", database "mydatabase", SSL off
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase sslmode=require')
<connection object at 0x6a180>
>>> 
silviana:~ cheeming$ export PGSSLMODE=require
silviana:~ cheeming$ python2.4
Python 2.4.5 (#1, Jul 15 2008, 23:37:00) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase')
<connection object at 0x6a1d8>

As I tried out using the environment variable way, I realised that you can also change the data source name to include sslmode=require, the same like the PGSSLMODE environment variable. I have updated the console listing above to show that as well.

Oh by the way, I heard a rumour that psycopg 1 doesn't support SSL. I am not 100% sure about it. Its quite hard to find out for sure (as I have tried on google, maybe I am not RTFM-ing enough) other than just testing it. The only info I found indicating this is some guy complaining about ssl not being supported in Windows.

Looking at the psycopg source code shows that there is support for something called sslmode. And in the ChangeLog file, its stated as follows:

2004-01-10  Federico Di Gregorio  <fog@debian.org>

    * module.c (psyco_connect): added "sslmode" parameter.

There are more environment variables that you can tweak to change the behaviour of the libpq C library and its stated here.

Hope this is useful for future PostgreSQL bug hunters!

I have always been interested with visualisation of data. I like to generate graphs from data. I have some ideas in my head for a while but never got around to actually doing it. Google Finance has a really nice visualisation for tracking stock prices and foreign exchange. So since I am interested in tracking the foreign exchange of the local banks in Malaysia I decided to do a bit of hacking to make my own Google Finance-ish interface to the data. The graph that I render is actually the percentage of change as compared to the forex rate of 4.77.

I've been scraping the foreign exchange data from RHB Bank website for a while and wanted to plot the data into a graph. I looked around a bit for some open source graphing and data visualisation tools and found Flot to be quite nice and its developed by some Django and jQuery fans, so maybe I am biased. Flot is a pure Javascript plotting library.

So I have a cron script that will scrape data from the RHB Bank website and writes it into a CSV file. I used BeautifulSoup to easily extract the data out of the HTML. I setup another cron script that will use the data from the CSV file and it will generate the HTML that will contain all data that is needed by Flot to draw the graph.

Flot is pretty easy to use and the basic conventions are pretty good. Since its a pure Javascript library all the code is on the HTML page and you can check it out if you want. I was thinking of adding it to Github gist but I think its not necessary at the moment since the code is very specific to my own needs. The documentation and examples are pretty good and its quite easy to learn. I particularly like the autoscaleMargin property for configuring how much margin it should automatically add to the axes so that the min and max value for the data would not be rendered on the edge of the grid. I also particularly like how easy it is to do selection and zooming. Just a few lines of code and its done.

So why did I mention I used a bit of Django?

I was lazy to find (and learn and setup) another templating language and decided to use Django's templating engine instead since the setup of that server that I am using works with Django. The HTML file that is generated is rendered off a Django template file using render_to_string() and I just pipe the stdout to a file. The only unsavoury thing is that I need to setup the DJANGO_SETTINGS_MODULE=settings and the settings.py file has only the TEMPLATE_DIRS setting.

Okay, so the next step is to add some news feed with the forex data to put more context while looking at the data. I guess that would be another weekend and another blog post then.

I have been looking for a more elegant way to code and structure a Django view so that it will support rendering of responses encoded in either HTML or JSON.

I do this because I want to add some AJAXy usability to my HTML forms while retaining some backwards compatibility for non-Javascript enabled Web browsers.

First, lets look at a simple Django view:

def foo(request):
    if request.method == 'POST':
        form = Form1(request.POST)
        if form.is_valid():
            form.save()
            return HttpResponseRedirect('/foo/success/')
    else:
        form = Form1()
    return render_to_response('app1/template1.html', {'form': form})

I was lazy and did some hacky stuff like this to support JSON response.

def foo(request):
    if request.method == 'POST':
        form = Form1(request.POST)
        if form.is_valid():
            form.save()
            if request.POST.get('type', '') == 'json':
                data = {} # define your own data structure
                return HttpResponse(simplejson.dumps(data),
                                    mimetype='application/json')
            else:
                return HttpResponseRedirect('/foo/success/')
    else:
        form = Form1()
    return render_to_response('app1/template1.html', {'form': form})

And for the HTML form, I needed some Javascript (using jQuery) like this:

    $("#id_cmd_submit").click(function(){
        $.post("/foo/", {type: "json"},
            function(data) {
                alert(data);
            }, "json");
    });

I was thinking to myself that this type parameter is kinda of ugly and not necessary. The HTTP protocol actually has a Accept header that you can use to define what encoding the HTTP response from the server can be and this is automatically added when you use the $.post() from jQuery using the json type.

If you examine the HTTP headers in the Django view through request.META['HTTP_ACCEPT'] you will see values like this:

application/json, text/javascript, */*

So to make the hacky method a bit less hacky, I did this:

def foo(request):
    if request.method == 'POST':
        form = Form1(request.POST)
        if form.is_valid():
            form.save()
            if 'application/json' in request.META.get('HTTP_ACCEPT', '')
                data = {} # define your own data structure
                return HttpResponse(simplejson.dumps(data),
                                    mimetype='application/json')
            else:
                return HttpResponseRedirect('/foo/success/')
    else:
        form = Form1()
    return render_to_response('app1/template1.html', {'form': form})

And the Javascript now looks like this:

    $("#id_cmd_submit").click(function(){
        $.post("/foo/", {},
            function(data) {
                alert(data);
            }, "json");
    });

Of course the code is still not very pretty and there are better ways to structure it better. In fact, all I did was just to make the API more "conforming" to HTTP and thus the Javascript code slightly cleaner. The Python code is somewhat the same.

As for the Python code, I see that someone has tried to make this style a bit prettier but I am not too sure if it will fit with all my use cases and I don't like the additional complexity.

I did read a bit about Adrian Holovaty's take on this AJAXy stuff. Its definitely cleaner than my approach but I wish Django have better support for HTTP's Accept header since I like the approach of using the Accept header. I found a middleware in Django snippets that can abstract away the complexity a bit in a more elegant way.

There is a good writeup on REST worst practices by Jacob Kaplan-Moss which touches a bit on using the Accept header instead. Malcolum Tredinnick also though on this a bit in an article about content types on this blog.

So I guess I did not bark up the wrong tree. And surprisingly, those blog articles that I found were quite recent. At least I am not outdated.

I got Trac 0.11.2.1 up and running with Apache using FastCGI and then decided to try out Bitten. I've read about it before but never seen it in action or configured it before. I downloaded the 0.6 version (r567) of Bitten and realised that it still requires Clearsilver. Trac 0.11 is supposed to be using a new templating system. But its not so bad, its pretty easy to install Clearsilver. Check Ticket #248 for the Clearsilver issue.

Then after that I realised that it requires the WebTracAdmin plugin, I suppose there are some code dependencies. But it doesn't require the plugin to be enabled in the Trac setup. Having WebTracAdmin installed will allow you to see additional Bitten specific Admin options in Trac. Check the install instructions for more info.

Things look like its working well. Clicking around doesn't seem to trigger any issues. I read more about how Bitten works and I realise I need to create a build recipe in the Admin section. So I head off to create one, but Trac continually gives me an error about having issues with missing or invalid form token, which might be caused by cookies.

So I went ahead to troubleshoot this problem and realised that the form token that is rendered in the HTML form is different from the form token that is stored in the cookie. After a bit more digging, it seems everytime I load the Bitten form in Trac, 2 requests are made to the server.

So what is the 2nd requests? And why does it matter? A bit more digging reveals that Trac will generate a different form token for each requests and the offending request is for an invalid Javascript file (wikitoolbar.js).

Because Bitten is written based on pre-Trac-0.11 code base, I guess there are some missing stuff, which lead to the template generating the wrong path for static files, which lead to making erroneous requests and finally creating this buggy behaviour with the form token. There is a ticket already about something similar and I reported my findings as well.

A bit of templating hacking here and voila Bitten seems to be working for real now, at least that is what I thought until I get to running Bitten slaves. Only after doing all this work did I realise that there is a Bitten for Trac 0.11, an experimental branch. I wonder if everything is working as it should. I am currently happy with the hacked up setup and won't want to modify it for the time being.

A Bitten slave will do all the grunt work of running the builds. You need to write some XML files as instructions for the slave. I wanted to write the whole build script in the Bitten recipe XML DSL but it seems that is not the right way to go. The way the Bitten slave runs the shell commands make it hard to do so. So you'll actually have to write a shell script first to do all that work (it makes sense anyway) and use the Bitten recipe to invoke it.

The Bitten slave will capture all the output and result of the commands that you execute and send it back to Trac. From my testing it seems that the Bitten slave is quite sensitive, I had a script that output new lines and that seemed to have broken the build. I actually had to make my scripts output less crap to make it a bit more Bitten friendly.

All in all, it wasn't that hard to setup and I am pretty happy with the setup at the moment. I am so happy that it is open sourced and I can hack on it. I am still trying to figure out how to use virtualenv in this whole setup so that I can also test the setup of python related dependencies. That will be another experiment for another day and maybe another blog post, I suppose...

Almost 2 weeks ago I was in Vilnius to attend EuroPython 2008. It was my first time in Vilnius and first time to attend a conference about Python, which is the main programming language that I've been using for the past 2 years of work at Exoweb.

It was interesting to meet so many people that are also using Python everyday in their work and projects. In the conference, I met up with Greg and Tomasz, who's Greg's partner in crime in the software business. They are part of Enpoka that is doing outsource software development. I also met up with Tobias, who drove all the way from Tromsø to Vilnius so that he can fix his car in Riga, its more than 2000+ kilometres, pretty crazy!

I attended some talks on Jython and PyPy. I have to say I am pretty impressed with how far Jython has come (and only recently realised the state of development of the JVM in the dynamic languages world) and the grandiose plans for PyPy.

The day ended with Guido's keynote but it was done through video conferencing and I guess it wasn't to the best effect. It was regarding Python 3000 which I've heard at least twice so far.

The next day started with a talk on a tool to visualise relationships between Python objects and duplicate code finder a.k.a CloneDigger. I listened to a talk on LEAN software development by Steve Alexander of Canonical but it was a bit too vague to me, although I thought his approach of using photographed stickies as slides was cool. There were some good talks of Python's more advanced topics (descriptors) and the internals of some of the standard data structures we use everyday like lists, dicts and sets.

I attended the Cython talk but I was getting distracted hacking away with Greg on using greenlets with twisted. Both of us were not so happy with the deferred style of coding asynchronous servers in twisted and wanted to find a more manageable approach. We have a pretty simple prototype and I would want to apply the same idea to a larger code base .

The day ended with a keynote by Hans Rosling, who I admire quite a bit due to his great TED talk he gave. He is more impressive live and I truly enjoyed his talk although it wasn't much about Python. There were some lightning talks before the keynote and the one that I can remember very clearly is gSym, which is a python editor that visualises your code in different ways such as mathematical formulas and lisp style.

For the last day, one of the highlights was a talk about hacking your dna with the amazon cloud. I talked a bit with Mike the DNA hacker and learned quite a bit about his travels in South East Asia which were pretty cool. After the hacking on greenlets and twisted, Greg decided to hold an unconference session on it to discuss about it. Tommi, who knows quite a bit about twisted gave some good insights on the greenlet+twisted approach which is that although it works its dangerous.

There is a lot of material that I would like to catch up on but I haven't found the time to really dig into it. I really enjoyed the conference and thought it was a good chance to learn a few things and meet similar minded people.

I managed to meet briefly with some Django contributors and Jim Baker who's working hard pushing out Jython 2.5. He says he'll be in China in September, maybe he'll come over and give a talk on Python and Jython. I am kinda interested in Jython myself and have been submitting some bug reports, hopefully its useful for them . I was not really ready for the Django sprint and spent a bit of time looking into Django new forms admin. At least I learnt a bit about the new branch that will go into 1.0 soon.

I have bookmarked some of the more interesting stuff I learned from the conference in del.icio.us and also took some pictures while sightseeing.

I have more to say, ok maybe not. Over!

Leave a comment