What’s new in Python 2.6

Full release notes are here. Here are something that catch my eyes.

  • with statement with is a statement that simplifies the task of elegantly acquiring and releasing resources, and moves the task of handling exception to the resource manager author. For example, the file object now supports with statement, and when using this statement, the job of closing the file on regularly releasing the object or on an exception will be the task of the file object itself. It also makes locking/unlocking a resource amazingly intuitive, by moving the real (and repeated) job to the resource manager author.
  • per-user site-packages. Without this, I have always been making my own per-user site-pacakages. Handling different versions of packages for different Python versions is not an easy task. Now it’s automatically done. No need to sudo when installing a package!
  • multiprocessing package. I have had some very bad experience with Python’s threading module. Python threading is useless in many cases when your code enters a chunk of time-consuming C code: even when the C code is blocked by I/O, the other Python threads will not be waken up if the C code is not Python-aware. This issue is made worse by the infamous GIL, which means your program will always be executed serially no matter how nice your multi-threading code skill is or how many CPUs you have. Multiprocessing is way out of this, though it consumes a bit more resources. It also has the advantage of being transparently distributed: you can run multiple processes across a cluster of computer very easily.
  • Advanced string formatting. Now you can have many nice features found in Django‘s template engine.
  • print is now a function. This allows you disable all prints in your program by redefining it. I was typically too lazy to use the logging module for debugging purpose, and now this change to print saves me some time from removing all debugging print statements. While, logging is still better.
  • New I/O library, which looks so similar to Java’s I/O library.
  • Abstract Classes. Now if you want to build some extensible/customizable software, you have an easy way to specify which methods must be defined instead of just saying it in the document.
  • fractions module. I like Python’s support for long integer, decimal and now fraction data type. double is evil in many cases despite its efficiency.
  • json module is now part of Python standard library.
  • Now both md5 and sha modules have been removed. Use hashlib instead.

CentOS differs from many other distros by enabling root account during setup. I prefer the Ubuntu’s (and OS X’s) way of using a separate admin account and having root account disabled. When there is a need to perform administrative task, just run the command with sudo and easily prevent the risk of abusing root privileges and doing stupid things. Following this guide, I was able to make this work on CentOS.

  1. First, log in as root account. You can switch to root account from any account by running su and typing the root password.
  2. Enabling sudo. If you are not comfortable with vim, run

    export EDITOR=gedit

    first. Now run


    The lines starting with # are comment lines and will be ignored. Just uncomment the following line:

    # %wheel  ALL=(ALL)       ALL

    by removing the # at the beginning. This line means that anybody in the group wheel can use sudo to run anything from anywhere.

  3. Add an account to group wheel. For example, if the account you use to perform administrative task is isteering, run

    gpasswd -a isteering wheel

    Now you can sudo from user isteering

  4. Disable root account. This is done by running passwd to lock the account:

    passwd -l root

It is quite obvious after we perform the above steps, we have just created a second root account: the user isteering is exactly the same as root user, just having a different name. So we have not added much protection, if the attacker can guess the name of this new account. So you might want to consider limiting where the user can log in from. Use your favorite editor to edit file /etc/security/access.conf. Add the following lines for the admin group:

-:wheel:ALL EXCEPT LOCAL 192.168.1.

This will deny user in group wheel to log in from anywhere but 192.168.1. subnetwork (note the suffix dot) or host You still need to add this line

auth       required     pam_access.so

to /etc/pam.d/sshd to tell SSH server to consult the access control, otherwise SSH server by default will ignore this access control mechanism built in PAM.

In developer-friendly versus customer-friendly, the author made a really good point about Apple’s controversial stance in many of its recent decisions.

A follow-up made it even clearer using OS X application development as an example. Though I still prefer to code in an environment that is developer friendly, which I think both Linux and OS X are (at least for my quick prototyping needs), for real-life applications I depend on, I’d appreciate quality of Apple products.

I am not a big fan of any Redhat or derivative distributions, but this time I was forced to use CentOS because the RAID card of the server is only well supported by RHEL and (therefore) CentOS. As a newbie of RPM and yum, my way of doing things can be quite stupid. So I’d appreciate your comments and corrections.

First, the default Java environment on CentOS is GIJ, as are most Linux Distros. So the first thing to do is to get official Java 6 installed. Though there are tutorials creating RPM package from Sun’s distributed files, I am just setting up this one machine and will just install Sun’s binary distribution directly.

  • Download the binary distribution from Sun’s website: http://java.sun.com/javase/downloads/index.jsp. Make sure you download the “self-extracting file” for “Linux Platform”.
  • You can also copy the URL to the download file, and use `wget’ to download from command line. What I did was:
    cd ~/Desktop
    wget http://www.java.net/download/jdk6/6u10/promoted/b32/binaries/jre-6u10-rc2-bin-b32-linux-i586-12_sep_2008.bin
  • Now run that installer file. Before you run it, you must tell Linux it is safe to run it, and then run it:
    chmod u+x jre-6u10-rc2-bin-b32-linux-i586-12_sep_2008.bin
  • This just extracts the JRE into the current folder. We need to move it to a permanent location of your choice. Any location would work. I chose /usr/lib. So I did:
    mv jre1.6.0_10 /usr/lib/
  • Because we might update to a new JRE in the future, and we hate to change all things that are dependent on JRE location after that update (think about change 20 JAVA_HOME settings after each JRE update), we will set up a symbolic link (shortcut in Windows vocabulary) for JRE. In the future, when we update JRE, we just update that link to get all other programs to use the new JRE:
    cd /usr/lib
    ln -s jre1.6.0_10 jre
  • In the future, when we need to set JAVA_HOME, we will set it to the link /usr/lib/jre, which always points to the latest JRE. So when getting a new JRE, we just update this link, and do not need to change JAVA_HOME for each programs we use. You can also set up a global JAVA_HOME for programs that respect it:
    echo "export JAVA_HOME=/usr/lib/jre" >> ~/.bashrc

    The above command will append one export line bash‘s configuration, which bash will read every time it starts. You may also do

    echo "export PATH=$JAVA_HOME/bin:$PATH" >> ~/.bashrc

    so that every time you type java on the command line it will use the latest Java.

Now it’s time to download and set Tomcat in almost the exact same way:

  • Download binary distribution of Tomcat from http://tomcat.apache.org/download-55.cgi. I would choose the .tar.gz file (also known as tarball):
    cd ~/Desktop
    wget http://download.nextag.com/apache/tomcat/tomcat-5/v5.5.27/bin/apache-tomcat-5.5.27.tar.gz
  • Now decompress it. The way to decompress a tarball is almost always:
    tar -xzf apache-tomcat-5.5.27.tar.gz

    This will give you a new folder (named apache-tomcat-5.5.27 in my case) that contains the tomcat program.

  • Now move the new folder to a permanent location. As for the JRE above, any location would work. I chose /opt, where people typically use to store relatively independent programs:
    mv apache-tomcat-5.5.27 /opt/tomcat-5.5

    This command moves the new tomcat folder to /opt and renames it to tomcat-5.5.

  • Similar to setting up JAVA_HOME, we will also need to set up CATALINA_HOME. So do:
    echo "export CATALINA_HOME=/opt/tomcat-5.5" >> ~/.bashrc
    echo "CATALINA_BASE=/opt/tomcat-5.5"  >> ~/.bashrc

That’s it. Close the terminal window and start a new one so that all new settings get loaded. You should now be able to start Tomcat by running familiar things like /opt/tomcat-5.5/bin/catalina.sh start


Daemonize python script

In developing a new feature in ChemMine, I encountered the problem of daemonizing a python script, for a second time.

Typically a HTTP request can be handled pretty fast. So unless I need to call some C or C-based code that can potentially crash the server, I would just process it inside the server process. For these C and C-based code, I would call it using os.system(...). But when the external process takes very long time, the python-based application server will wait and wait, and finally the client will timeout the request.

That’s why I need to daemonize these time-consuming processes, or in other words, run them in the “background” so that the application server can respond the client ASAP. Daemonizing them looks easy at the beginning, and there is a recipe for that purpose. The thing is that it makes it really hard to debug. Sometimes when you test it inside a terminal, it returns instantly just as it is successfully daemonized; but when you test it in the application server you would find the server will still wait for it to finish. So here is the most important thing to remember when you daemonize a process: close ALL fd’s at the OS level, that is, call os.close(...) to close fd from 0 up to 1023.

That would also makes it impossible to debug, because the standard output and error are also closed, and all error messages are discarded. So I would leave the standard error to open, until I iron out all the bugs.

Also, if you call further external programs in this daemonized processing using services like os.system(...), make sure you redirect their standard output and standard error. These are not available in the daemonized process, and the spawned process from it will not be able to inherit them and can potentially die when it fails to write something.

Budget cut

The queue at DMV is so damn long. There is not even enough in door space, and people have to stand under the bitter sunshine waiting to be let inside.

That’s the effect of budget cut, which leads to the lay-off of temp workers and this insanely long queue which I am standing in…

Update: now I was told to come back next week because they are closed. Ugh. I guess I will have to get used to this and be ready for more…

Update 2: I checked the news and found that 

Gov. Arnold Schwarzenegger ordered the layoffs of thousands of state workers Thursday and steep pay cuts for most other state employees to ease California’s budget crunch, moves that could mean longer lines at the DMV and delays in processing of workers’ compensation claims.


Also on Thursday, about 10,300 seasonal and temporary state workers were given layoff notices.

<<news link>>

On-Screen Keyboard

I wanna surf the web on my TV, sitting in my couch. However, I hate to carry a full-size keyboard. Maybe a remote control is the ultimate choice, but a small wireless mouse plus a on-screen keyboard is the easiest to implement.

GOK is the first thing I found. However, it was too “advanced” that I could not figure out how to use it for my simple application. A post on the Ubuntu forum showed that it confused more than just me. Luckily there are alternatives like SOK and the old but simple xvkbd.