depthq - ccm|mac

Plans and stuff

I haven't written in a long while. It is almost going to be a year since my last blog post and as I type this I don't really know what to write about.

Hmmm, how about some future plans?

Initially I thought that I would just treat this blog simply as a brain dump. But I have found that it is not really useful for anyone except for myself. So I would like to split this blog into 2 separate ones. One for just software development and other technical mumbo jumbo and another for random thoughts.

So I have a bunch of stuff I would need to do before I can move forward with my new plans. Gotta upgrade this blog engine I am using or change to another one altogether. And probably need to find a nice and simple theme. Gotta make it more updated since it looks quite dated at the moment.

OK, thats all for now. Happy Merdeka fellow Malaysians and Happy 10th anniversary Exoweb!

Original post blogged on b2evolution.

A minor update after 6 months absence

I haven't written for about 6 months. My mind is just filled with so many things that I forgot that I have a place to actually vent and rant properly (other than my twitter but only 140 characters at a time).

Quite a bit has happened since.

I am working on my startup and we're constantly trying different ideas. Life is hectic and hard to predict but it can get pretty exciting like a roller coaster ride. We've raised some money and recently participated in Techcrunch50 in San Francisco. We met a lot of influential and interesting people and hope we can make our new found connections work well for us.

I feel that I've changed my view a bit about the Malaysian life in these 9 months of being back. I guess I've been poisoned by Yasmin's work. I loved Sepet, Talentime and Mukhsin. Gubra's story was a bit too dark for me. Her work reminds me of my childhood and the beauty of life here. Nothing is perfect but maybe that is beauty in itself. Oh I am getting mellow...

... listening to Okuribito soundtrack as I write ...

In my free time, I try to run and cycle a bit. I've done a bunch of quarter and half marathons. Slowly getting better each time but still far off from my target. I still can't imagine doing a full marathon yet. Running really sucks out a lot of your mental and physical energy but it also, as my close friend says, "never takes more than it gives".

I have this minor obsession with bicycles and cameras. Sometimes I can surf endlessly just reading and learning about cameras and bicycles. My current fad is looking into Russian cameras, like FED and Kiev. They are basically Leica and Contax clones.

I recently bought a FED Micron from Ebay. It is an interesting camera due to it taking half frame pictures from 35mm film. But my first roll of film was a total disaster. I think the film wasn't setup properly and only got about 10-20 shots out of 72 shots. I'll probably try again but playing with film is not as cheap as digital. My other two targets are Russian rangefinders, namely FED 2 and Kiev 4.

My recent visit to San Francisco has reignited my passion for cycling. My long distance bike touring trip (3 months on the road?) is still a naive idea in my mind. Or maybe a month long backpacking trip to quench my thirst for some travel adventure. I wish I can find the time and energy to do it in the future when I am less busy and committed to my work.

Enough writing English, need to write some Python codes now

Original post blogged on b2evolution.

Make svnserve support multiple SVN users for one SSH account

I am using A Small Orange to host some of my stuff and I wanted to setup Subversion (SVN) to work there. It was pretty simple since this hosting company supports SVN through svnserve. How cool is that?

But the problem that I encountered with this is that I wanted to support more than one user. In the beginning I was thinking that I needed to share the single username and password with everyone that is working on the same SVN repository (or share a single private key). Oh the security implications that would bring!

But I soon realised that with svnserve and some SSH config tricks I can support multiple users on the repository. And the good thing is that I don't have to mange these damn passwords. I always preferred password-less setups with key pairs (many thanks to all the work done by people to make PKI possible).
And another good advantage is that you can restrict the user that is logging through SSH to only have access to some specific directory.

I found the details for this here (I think the A Small Orange community is really helpful). The only thing to look out for is that in Ubuntu 8.10 when you run the following command:

ssh-keygen -t dsa -f dev-dsa -P '' -C developer001

The password is not actually empty, it would be better to drop the -P and then enter an empty password in the input, instead of entering empty string at the command line. The same command worked fine in my Mac OSX 10.5.6.

And finally the new URL to access the repository is slightly different, it will be relative to the virtual directory that you point each user in your authorized_keys file to.

I actually use git svn quite a lot these days and I haven't really tested this with my git svn setup because the URL to the path of the SVN repository has changed and I am not sure what implications it would have on the git svn repository that is still referring to the old URL. I am keeping the old and new style for now until I figure things out.

I just wanted to share that although hosting a SVN repo behind Apache is probably one of the most popular ways to deploy Subversion repositories but I think this method has some distinct advantages and its quite flexible if you want finer grain control of security options, permissions and access controls.

Hope this hint help others out there and if anyone has done some work with git svn and changing of svn remote URLs, then it would be nice to hear from you.

Original post blogged on b2evolution.

Checking PostgreSQL to ensure it works with SSL or non-SSL ports

If you want to check to ensure your SSL or non-SSL ports in PostgreSQL are working properly, read further to find out how.

For SSL support, you need to setup your PostgreSQL to work with SSL properly. Just turn on the ssl flag in postgresql.conf and then make sure you've setup the server.crt as mentioned in PostgreSQL docs about Secure TCP/IP Connections with SSL.

If you want force psql to use non-SSL for the connection, you need to set the PGSSLMODE environment variable. If you're using bash, do something like this:

export PGSSLMODE=disable

And then try to run the psql to connect to the server. I would suggest that you be explicit in your pg_hba.conf to specify that you want hostnossl, so that the server will not accept SSL connections.

If you got things configured correctly, it should work without problems. On the other hand, you can force it to use SSL, like this:

export PGSSLMODE=require

You should see a fatal error message from psql, stating that there is no pg_hba.conf entry for SSL on. You can use the same method to test for SSL-only connections by using hostssl instead in pg_hba.conf.

This environment variable works not only for psql but also for any library or tool that uses libpq. For example, I include the console output for testing the psycopg library. I have already set up the PostgreSQL to work with SSL only.

silviana:~ cheeming$ export PGSSLMODE=disable
silviana:~ cheeming$ python2.4
Python 2.4.5 (#1, Jul 15 2008, 23:37:00) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg.__version__
'1.1.21'
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
psycopg.OperationalError: FATAL:  no pg_hba.conf entry for host "127.0.0.1", 
    user "cheeming", database "mydatabase", SSL off
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase sslmode=require')
<connection object at 0x6a180>
>>> 
silviana:~ cheeming$ export PGSSLMODE=require
silviana:~ cheeming$ python2.4
Python 2.4.5 (#1, Jul 15 2008, 23:37:00) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg.connect('host=127.0.0.1 dbname=mydatabase')
<connection object at 0x6a1d8>

As I tried out using the environment variable way, I realised that you can also change the data source name to include sslmode=require, the same like the PGSSLMODE environment variable. I have updated the console listing above to show that as well.

Oh by the way, I heard a rumour that psycopg 1 doesn't support SSL. I am not 100% sure about it. Its quite hard to find out for sure (as I have tried on google, maybe I am not RTFM-ing enough) other than just testing it. The only info I found indicating this is some guy complaining about ssl not being supported in Windows.

Looking at the psycopg source code shows that there is support for something called sslmode. And in the ChangeLog file, its stated as follows:

2004-01-10  Federico Di Gregorio  <fog@debian.org>

    * module.c (psyco_connect): added "sslmode" parameter.

There are more environment variables that you can tweak to change the behaviour of the libpq C library and its stated here.

Hope this is useful for future PostgreSQL bug hunters!

Original post blogged on b2evolution.

Some hints on integrating with Worldpay

I just want to write a short piece to document some of my experiences in working with the Worldpay payment system.

It is pretty similar to Paypal, you just need to POST a bunch of HTML form variables to a URL and its done. Worldpay has an option to send a Payment Response back to your server. Using this response you can do some integration with your backend to identify that payments have been made.

For Worldpay, it works for both single payment transactions or recurring payment transactions. To support this, Worldpay is flexible enough to not charge for the initial transaction and it acts more like a registration step, which is suitable for recurring payment. You can read more about it in Worldpay FuturePay's documentation on Regular Agreement Options.

A lot of information is passed back but I think the following are the most important:

transStatus
cartId
futurePayId
amount

If the transaction deducts money, it will include a transId.

If the transaction is a cancellation, the fields that are important are futurePayStatusChange and futurePayId.

There is a big list of fields that might be returned in the Payment Response and its documented here.

If the user selected a different currency as you initially set, it will be indicated in the group of variables with the authXXXX convention. Worldpay has also written a bit about their Exchange Rates and how it works.

To fully support the Worldpay FuturePay with dynamic recurring amounts or payment dates, there is a Remote Administration Interface (RAI) that you use programatically. But do note that you need an installation id that is specifically setup for RAI.

There are some security recommendations by Worldpay such as ensure the request is from Worldpay, MD5 security and make transactions more time dependent.

One thing that bugged me is how user defined variables are passed back to the Payment Response. The recurring and cancelled Payment Response would never include user defined variables because they don't store it in their system, as indicate here. So you'll have to create a system that would need to work in a more flexible way for those cancellation and recurring Payment Responses. I suppose this makes the system a bit more robust, but I am lazy and wish that recurring Payment Responses could send me back the user defined variables. I would view that as more consistent and simpler API to use.

Oh yea and finally, you can have a Shopper Response that will be shown to the user once the payment (or registration) is done. The Shopper Response is generated by the same callback that handles the Payment Response but it works slightly differently. It will take the HTML that you generate and then render it as part of their final payment confirmation page. The URL of that page is not the callback that handles the Payment Response but their own URL. Its like they are doing a copy and paste. But the copy is done by reading off the HTML that you generate by the Payment Response callback. Also there are some restrictions because you need to include a compulsory banner code, which will be expanded (or filled in) by Worldpay to include more details.

Worldpay has written some documentation on creating a Shopper Response but it took a while and a lot of reading to understand how it roughly works. I am still scratching my head over it and sort of given up. There is still whole interface that you can use to configure the Shopper Response page and change all the styles and colours but I think its just too damn difficult at the moment and I've kind of lost interest in doing further for now.

Okay I think that is all for now until I figure this Shopper Response crap...

Original post blogged on b2evolution.

Tracking Forex with Python and Flot (and also a bit of Django)

I have always been interested with visualisation of data. I like to generate graphs from data. I have some ideas in my head for a while but never got around to actually doing it. Google Finance has a really nice visualisation for tracking stock prices and foreign exchange. So since I am interested in tracking the foreign exchange of the local banks in Malaysia I decided to do a bit of hacking to make my own Google Finance-ish interface to the data. The graph that I render is actually the percentage of change as compared to the forex rate of 4.77.

I've been scraping the foreign exchange data from RHB Bank website for a while and wanted to plot the data into a graph. I looked around a bit for some open source graphing and data visualisation tools and found Flot to be quite nice and its developed by some Django and jQuery fans, so maybe I am biased. Flot is a pure Javascript plotting library.

So I have a cron script that will scrape data from the RHB Bank website and writes it into a CSV file. I used BeautifulSoup to easily extract the data out of the HTML. I setup another cron script that will use the data from the CSV file and it will generate the HTML that will contain all data that is needed by Flot to draw the graph.

Flot is pretty easy to use and the basic conventions are pretty good. Since its a pure Javascript library all the code is on the HTML page and you can check it out if you want. I was thinking of adding it to Github gist but I think its not necessary at the moment since the code is very specific to my own needs. The documentation and examples are pretty good and its quite easy to learn. I particularly like the autoscaleMargin property for configuring how much margin it should automatically add to the axes so that the min and max value for the data would not be rendered on the edge of the grid. I also particularly like how easy it is to do selection and zooming. Just a few lines of code and its done.

So why did I mention I used a bit of Django?

I was lazy to find (and learn and setup) another templating language and decided to use Django's templating engine instead since the setup of that server that I am using works with Django. The HTML file that is generated is rendered off a Django template file using render_to_string() and I just pipe the stdout to a file. The only unsavoury thing is that I need to setup the DJANGO_SETTINGS_MODULE=settings and the settings.py file has only the TEMPLATE_DIRS setting.

Okay, so the next step is to add some news feed with the forex data to put more context while looking at the data. I guess that would be another weekend and another blog post then.

Original post blogged on b2evolution.

Make your own album cover meme

I read this initially from Yoon Kit's blog post.

I am lazy to repost the rules, you can check it out above.

I am no artist, so here it goes:

Not too bad I guess...

Details:

The random wiki article (band name): http://en.wikipedia.org/wiki/Quebec_Autoroute_10
The random flickr picture (album art): http://www.flickr.com/photos/26637167@N08/3326019488/
The random quote (album name): http://www.quotationspage.com/quote/33751.html

Original post blogged on b2evolution.

Hacking Ubiquity for better Identi.ca/Twitter usage

I've been using Ubiquity plugin with Firefox for quite a while, mostly for Tinyurl-ing and Identi.ca-ing/Twitter-ing.

For those who don't know what it is: it is a Quicksilver-ish clone but for Firefox and it allows you to create shortcuts for a lot of the web related tasks that you might normally perform. For example: converting a URL to tinyurl, you just need to hit a shortcut, type a command and voila, its done. No need to copy and paste or click on some button. Or send a quick tweet; no need to switch application or create a new tab and login to twitter.com.

And for those who don't know what is Quicksilver: its a very cool program launcher for the OSX and it actually does a lot more than a simple program launcher, although I don't use much of the other, more powerful functions.

Ubiquity is a pretty good idea and quite good to use, minus the bug #19 (broken growl notification) that drives me crazy, but I am getting used to it.

Since I am a programmer I guess I should hack on more code to make my life easier (Hmmm, maybe I should fix that bug #19). I've been using Identi.ca to post my tweets and because I was lazy I wanted to do it in Ubiquity and I got a Ubiquity script from the Ubiquity Firefox Google Groups that would do the trick.

I've been using it for quite a while and realised I have the tendency to make a mistake when replying to tweets (using @nickname) in Ubiquity. So I thought that it would be useful to change the script to make the experience slightly better.

Before I start, I would like to just talk about my micro-blog setup: I have setup my Identi.ca to post my tweets to Twitter. And my normal use case is to tweet using Identi.ca and then check the feed with all those that I following in Twitter since I have more activity there. So if I see anything interesting that I would like to reply to or retweet I can just type in the username in my status message. But the problem I have is that I might type wrongly so I just wanted a way to make it less error prone.

So what I did was to hack the original Identi.ca script and added a simple check when I type the @ symbol in my message. It will extract all the Twitter usernames from the current active webpage when I activated the Ubiquity command prompt and then display in the Ubiquity preview panel as a list of candidates.

As I slowly complete the nickname, it will drill down the list of candidates based on the character by character matching. If I make a typo, then there would be no candidate names in the list. Its just a very simple hack but I think it would make my replying a bit less error prone. The next step would be to have an easy way to match the reply to the correct tweet. I wonder if there is a way to do it in my case, since I am using Identi.ca. I haven't really explored the API at all.

Because of this small itch I managed to learn a bit about hacking Ubiquity and I have to say that its really straightforward and the fact that I can use jQuery makes programming in Javascript almost fun. I would rather code in Python but I think I can live with this.

So if anyone wants to try out the hack I made, the code is hosted as a Gist in GitHub. Do give me some feedback, whether you find it useful or useless.

Original post blogged on b2evolution.