depthq - ccm|mac

I haven't written for about 6 months. My mind is just filled with so many things that I forgot that I have a place to actually vent and rant properly (other than my twitter but only 140 characters at a time).

Quite a bit has happened since.

I am working on my startup and we're constantly trying different ideas. Life is hectic and hard to predict but it can get pretty exciting like a roller coaster ride. We've raised some money and recently participated in Techcrunch50 in San Francisco. We met a lot of influential and interesting people and hope we can make our new found connections work well for us.

I feel that I've changed my view a bit about the Malaysian life in these 9 months of being back. I guess I've been poisoned by Yasmin's work. I loved Sepet, Talentime and Mukhsin. Gubra's story was a bit too dark for me. Her work reminds me of my childhood and the beauty of life here. Nothing is perfect but maybe that is beauty in itself. Oh I am getting mellow...

... listening to Okuribito soundtrack as I write ...

In my free time, I try to run and cycle a bit. I've done a bunch of quarter and half marathons. Slowly getting better each time but still far off from my target. I still can't imagine doing a full marathon yet. Running really sucks out a lot of your mental and physical energy but it also, as my close friend says, "never takes more than it gives".

I have this minor obsession with bicycles and cameras. Sometimes I can surf endlessly just reading and learning about cameras and bicycles. My current fad is looking into Russian cameras, like FED and Kiev. They are basically Leica and Contax clones.

I recently bought a FED Micron from Ebay. It is an interesting camera due to it taking half frame pictures from 35mm film. But my first roll of film was a total disaster. I think the film wasn't setup properly and only got about 10-20 shots out of 72 shots. I'll probably try again but playing with film is not as cheap as digital. My other two targets are Russian rangefinders, namely FED 2 and Kiev 4.

My recent visit to San Francisco has reignited my passion for cycling. My long distance bike touring trip (3 months on the road?) is still a naive idea in my mind. Or maybe a month long backpacking trip to quench my thirst for some travel adventure. I wish I can find the time and energy to do it in the future when I am less busy and committed to my work.

Enough writing English, need to write some Python codes now

I am using A Small Orange to host some of my stuff and I wanted to setup Subversion (SVN) to work there. It was pretty simple since this hosting company supports SVN through svnserve. How cool is that?

But the problem that I encountered with this is that I wanted to support more than one user. In the beginning I was thinking that I needed to share the single username and password with everyone that is working on the same SVN repository (or share a single private key). Oh the security implications that would bring!

But I soon realised that with svnserve and some SSH config tricks I can support multiple users on the repository. And the good thing is that I don't have to mange these damn passwords. I always preferred password-less setups with key pairs (many thanks to all the work done by people to make PKI possible).
And another good advantage is that you can restrict the user that is logging through SSH to only have access to some specific directory.

I found the details for this here (I think the A Small Orange community is really helpful). The only thing to look out for is that in Ubuntu 8.10 when you run the following command:

ssh-keygen -t dsa -f dev-dsa -P '' -C developer001

The password is not actually empty, it would be better to drop the -P and then enter an empty password in the input, instead of entering empty string at the command line. The same command worked fine in my Mac OSX 10.5.6.

And finally the new URL to access the repository is slightly different, it will be relative to the virtual directory that you point each user in your authorized_keys file to.

I actually use git svn quite a lot these days and I haven't really tested this with my git svn setup because the URL to the path of the SVN repository has changed and I am not sure what implications it would have on the git svn repository that is still referring to the old URL. I am keeping the old and new style for now until I figure things out.

I just wanted to share that although hosting a SVN repo behind Apache is probably one of the most popular ways to deploy Subversion repositories but I think this method has some distinct advantages and its quite flexible if you want finer grain control of security options, permissions and access controls.

Hope this hint help others out there and if anyone has done some work with git svn and changing of svn remote URLs, then it would be nice to hear from you.

If you want to check to ensure your SSL or non-SSL ports in PostgreSQL are working properly, read further to find out how.

For SSL support, you need to setup your PostgreSQL to work with SSL properly. Just turn on the ssl flag in postgresql.conf and then make sure you've setup the server.crt as mentioned in PostgreSQL docs about Secure TCP/IP Connections with SSL.

If you want force psql to use non-SSL for the connection, you need to set the PGSSLMODE environment variable. If you're using bash, do something like this:

export PGSSLMODE=disable

And then try to run the psql to connect to the server. I would suggest that you be explicit in your pg_hba.conf to specify that you want hostnossl, so that the server will not accept SSL connections.

If you got things configured correctly, it should work without problems. On the other hand, you can force it to use SSL, like this:

export PGSSLMODE=require

You should see a fatal error message from psql, stating that there is no pg_hba.conf entry for SSL on. You can use the same method to test for SSL-only connections by using hostssl instead in pg_hba.conf.

This environment variable works not only for psql but also for any library or tool that uses libpq. For example, I include the console output for testing the psycopg library. I have already set up the PostgreSQL to work with SSL only.

silviana:~ cheeming$ export PGSSLMODE=disable
silviana:~ cheeming$ python2.4
Python 2.4.5 (#1, Jul 15 2008, 23:37:00) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg.__version__
'1.1.21'
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
psycopg.OperationalError: FATAL:  no pg_hba.conf entry for host "127.0.0.1", 
    user "cheeming", database "mydatabase", SSL off
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase sslmode=require')
<connection object at 0x6a180>
>>> 
silviana:~ cheeming$ export PGSSLMODE=require
silviana:~ cheeming$ python2.4
Python 2.4.5 (#1, Jul 15 2008, 23:37:00) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase')
<connection object at 0x6a1d8>

As I tried out using the environment variable way, I realised that you can also change the data source name to include sslmode=require, the same like the PGSSLMODE environment variable. I have updated the console listing above to show that as well.

Oh by the way, I heard a rumour that psycopg 1 doesn't support SSL. I am not 100% sure about it. Its quite hard to find out for sure (as I have tried on google, maybe I am not RTFM-ing enough) other than just testing it. The only info I found indicating this is some guy complaining about ssl not being supported in Windows.

Looking at the psycopg source code shows that there is support for something called sslmode. And in the ChangeLog file, its stated as follows:

2004-01-10  Federico Di Gregorio  <fog@debian.org>

    * module.c (psyco_connect): added "sslmode" parameter.

There are more environment variables that you can tweak to change the behaviour of the libpq C library and its stated here.

Hope this is useful for future PostgreSQL bug hunters!

I just want to write a short piece to document some of my experiences in working with the Worldpay payment system.

It is pretty similar to Paypal, you just need to POST a bunch of HTML form variables to a URL and its done. Worldpay has an option to send a Payment Response back to your server. Using this response you can do some integration with your backend to identify that payments have been made.

For Worldpay, it works for both single payment transactions or recurring payment transactions. To support this, Worldpay is flexible enough to not charge for the initial transaction and it acts more like a registration step, which is suitable for recurring payment. You can read more about it in Worldpay FuturePay's documentation on Regular Agreement Options.

A lot of information is passed back but I think the following are the most important:

• transStatus
• cartId
• futurePayId
• amount

If the transaction deducts money, it will include a transId.

If the transaction is a cancellation, the fields that are important are futurePayStatusChange and futurePayId.

There is a big list of fields that might be returned in the Payment Response and its documented here.

If the user selected a different currency as you initially set, it will be indicated in the group of variables with the authXXXX convention. Worldpay has also written a bit about their Exchange Rates and how it works.

To fully support the Worldpay FuturePay with dynamic recurring amounts or payment dates, there is a Remote Administration Interface (RAI) that you use programatically. But do note that you need an installation id that is specifically setup for RAI.

There are some security recommendations by Worldpay such as ensure the request is from Worldpay, MD5 security and make transactions more time dependent.

One thing that bugged me is how user defined variables are passed back to the Payment Response. The recurring and cancelled Payment Response would never include user defined variables because they don't store it in their system, as indicate here. So you'll have to create a system that would need to work in a more flexible way for those cancellation and recurring Payment Responses. I suppose this makes the system a bit more robust, but I am lazy and wish that recurring Payment Responses could send me back the user defined variables. I would view that as more consistent and simpler API to use.

Oh yea and finally, you can have a Shopper Response that will be shown to the user once the payment (or registration) is done. The Shopper Response is generated by the same callback that handles the Payment Response but it works slightly differently. It will take the HTML that you generate and then render it as part of their final payment confirmation page. The URL of that page is not the callback that handles the Payment Response but their own URL. Its like they are doing a copy and paste. But the copy is done by reading off the HTML that you generate by the Payment Response callback. Also there are some restrictions because you need to include a compulsory banner code, which will be expanded (or filled in) by Worldpay to include more details.

Worldpay has written some documentation on creating a Shopper Response but it took a while and a lot of reading to understand how it roughly works. I am still scratching my head over it and sort of given up. There is still whole interface that you can use to configure the Shopper Response page and change all the styles and colours but I think its just too damn difficult at the moment and I've kind of lost interest in doing further for now.

Okay I think that is all for now until I figure this Shopper Response crap...

I have always been interested with visualisation of data. I like to generate graphs from data. I have some ideas in my head for a while but never got around to actually doing it. Google Finance has a really nice visualisation for tracking stock prices and foreign exchange. So since I am interested in tracking the foreign exchange of the local banks in Malaysia I decided to do a bit of hacking to make my own Google Finance-ish interface to the data. The graph that I render is actually the percentage of change as compared to the forex rate of 4.77.

I've been scraping the foreign exchange data from RHB Bank website for a while and wanted to plot the data into a graph. I looked around a bit for some open source graphing and data visualisation tools and found Flot to be quite nice and its developed by some Django and jQuery fans, so maybe I am biased. Flot is a pure Javascript plotting library.

So I have a cron script that will scrape data from the RHB Bank website and writes it into a CSV file. I used BeautifulSoup to easily extract the data out of the HTML. I setup another cron script that will use the data from the CSV file and it will generate the HTML that will contain all data that is needed by Flot to draw the graph.

Flot is pretty easy to use and the basic conventions are pretty good. Since its a pure Javascript library all the code is on the HTML page and you can check it out if you want. I was thinking of adding it to Github gist but I think its not necessary at the moment since the code is very specific to my own needs. The documentation and examples are pretty good and its quite easy to learn. I particularly like the autoscaleMargin property for configuring how much margin it should automatically add to the axes so that the min and max value for the data would not be rendered on the edge of the grid. I also particularly like how easy it is to do selection and zooming. Just a few lines of code and its done.

So why did I mention I used a bit of Django?

I was lazy to find (and learn and setup) another templating language and decided to use Django's templating engine instead since the setup of that server that I am using works with Django. The HTML file that is generated is rendered off a Django template file using render_to_string() and I just pipe the stdout to a file. The only unsavoury thing is that I need to setup the DJANGO_SETTINGS_MODULE=settings and the settings.py file has only the TEMPLATE_DIRS setting.

Okay, so the next step is to add some news feed with the forex data to put more context while looking at the data. I guess that would be another weekend and another blog post then.