Working with unicode in Python (again)

This time I have stumbled (again) a unicode problem using some Python code which was supposed to be perfectly suitable for doing this since it even started with


#!/usr/bin/env python
# -*- coding: UTF-8 -*-

It was quite some time since the last post, but this does not mean I haven’t done anything interesting :). It is just that it was so much interesting that I didn’t have any time to write anything.

Anyway, this time I have stumbled (again) a unicode problem using some Python code which was supposed to be perfectly suitable for doing this since it even started with


#!/usr/bin/env python
# -*- coding: UTF-8 -*-

It went perfectly fine when running in Eclipse, but to my huge surprise I’ve got problems when running the unit tests from the command line in terminal. Whaaat? It just worked!

Well, declaring your source as UTF-8 is not enough of course. There are several things to check when getting the “UnicodeDecodeError: ‘ascii’ codec can’t decode byte … in position …: ordinal not in range(128)”-kind of errors. Googling around didn’t bring me much luck to my surprise, so there is are my findings for the next time :).

First of all make absolutely sure you haven’t forgotten the ‘u’ character before your strings containing the unicode strings. Yep, just like that you screw up the rest of the unicode support. Python (ok, I admit, I use 2.5.4) treats a ‘string’ as a regular string and not as a unicode. So write u’string’ instead!

Second, when doing things file operations don’t forget that you don’t get the unicode by default. Consider the following:


message = u'unicode message'
file_handle.write(message)

Well, guess what. You get a problem when writing the string away. It cannot be recognized. So the solution would be to do something like this


encoded_message = message.encode(u"utf-8")
file_handle.write(encoded_message)

But that’s only a half of the problem. At some point you will be reading this data back. And most probably you would like to get your beloved unicode thingy back. Just doing the following will hardly help:


file_handle = open(full_name, 'r')
line = file_handle.readline()

The following will save your day:


file_handle = open(full_name, 'r')
line = file_handle.readline().decode('utf-8')

Voila. I hope this saves some frustration to somebody, even if it will be me some month later :).

Enjoy!


Comments from Andy (thanks!):

Probably there is nothing new for you in what I am saying below, however, from my experience, it covers 99% unicode-related errors.

Unicode string is s sequence of code points in range 0 to 0x10ffff. Encoding is a way of serializing this sequence, so thay can be represented in memory, written to a file, sent over a socket etc.
Encoding unicode string is needed _at least_ because its ‘as is’ byte representation is not portable due to byte order issues.

It is _recommended_ that you work with unicode string internally provided the language/API supports unicode
It is _must_ that you encode the string to be consumed by another program.

Subversion 1.6.3 on Ubuntu 9.02

Since installing subclipse I’ve got same problem I’ve already had on MacOSX when my subversion command line client (1.5.x) started to complain that the repository was already used with a newer version. Downgrading repository version works, but not a very nice solution, so I have decided to get the latest version. Apparently this is not just an aptitude command, but a little more. This is what have worked for me finally:

Download the source code for 1.6.3 (or later when it becomes available) and the corresponding dependencies from http://subversion.tigris.org. You have to have the following files in the case of 1.6.3:

  • subversion-1.6.3.tar.gz
  • tar xvf subversion-deps-1.6.3.tar.gz

Download the OpenSSL from http://openssl.org/source/ if you need https support:

  • openssl-0.9.8k.tar.gz

Then perform the following commands:

sudo apt-get install libssl-dev
sudo apt-get install zlib1g-dev


tar xvfz subversion-1.6.3.tar.gz
tar xvfz subversion-deps-1.6.3.tar.gz
tar xvfz openssl-0.9.8k.tar.gz


cd openssl-0.9.8k
./config
make


cd ../subversion-1.6.3
./configure --with-openssl=/usr/local/ssl --with-ssl --with-zlib=/usr/include --without-berkley-db
make
sudo make install

Having all this I’ve got working subversion (client) 1.6.3 under Ubuntu 9.02 .

TIP: If you done all above and typing svn --version still shows the default 1.5.x try another console. The installation of subversion apparently adds a new path which not picked up on the fly.

Chello (ISP) frustration

Yesterday was a nig eye-opener / frustrating night for me. After getting really fed up and even more convinced that my 30/7 kbps download/upload is not a common practice I have started with an investigation on who is doing this to my internet connection. Well, it took my laziness almost two years to get to this point, which also says how much I use my internet connection, but anyway…

First, I assumed it is me. What’s new, a techie that has more devices at home than braincells can definitely screw things up. I am one of those types. Here is the original setup:

  • Arris TM502B/220 cable modem provided to me generously by my ISP Chello (The Netherlands);
  • Linksys WRT54GL with Tomato formware (upgraded to 1.25 as one of the actions below);
  • Numerous devices eventually connecting to the router to get their (drops of) Internet connection.

The last drop was when a friend of mine who had presumably cheaper internet connection claimed higher upload speed than what I had. Let’s nail it down… Well, after hours of googling and changing the settings of my router’s firmware I’ve got zero result. Next step – hardware.

First, I have tried to connect to the modem directly excluding the router (which was my initial suspicion due to QoS settings and who-knows-what-else). This didn’t improve things even a bit. Still the same frustrating numbers (ses below).

Second, the most crusial step. Change the modem itself. Luckily I still have an old Motorola modem (no VoIP phone connection, but I don’t use it anyway). When I’ve got my new modem they didn’t take away the old one, so why not to try. This time luck was definitly on my side, since connecting the old modem worked like a charm, so additional setup was needed. Direct connection didn’t work though, so I had to connect to the router. So be it, it is the desired setup in any case. And then…

TADAA!!!

I run the speedtest.nl and I gt the numbers which raised my eyebrows in a huge frustration. Ok, first of myself that I have waited so long to get to this point, but second because of my ISP sending me crap. So here it is:

Setup with Arris modem, the results of speedtest.nl, (<laugh>):

  • Download 0.25 Mbps
  • Upload 0.06 Mpbps

(</laugh>)

With the old Motorola modem:

  • Download 15.57 Mbps
  • Upload 1.45 Mpbps

Feel the difference!

I immediately put back QoS to make sure I (or my wife :)) don’t eat up the whole bandwidth with one download and forced myself to put back the Arris modem back on the shelf carefully not breaking it in pieces.

Conclusions
The new modem (Arris) was cutting my connection ridiculously while the only added feature (phone connection) was not used by me at all. I will be cancelling my subscription(s) soon I think and reshaping my package to higher internet only. This costed me too much waiting time and frustrations to keep it like this for the last TWO YEARS!

p.s. Both modems provide basic information on http://192.168.100.1 and the old Motorola with firmware copyrighted in 2002 has even settings like enabling DHCP (which I don’t need and even don’t want) configurable, while Arris’es settings are kept behind the (unknown to me) password.

CherryPy + MySQLdb

After hours of searching and hitting the walls with the following error:

File "build/bdist.macosx-10.3-i386/egg/MySQLdb/__init__.py", line 19, in
File "build/bdist.macosx-10.3-i386/egg/_mysql.py", line 7, in
File "build/bdist.macosx-10.3-i386/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/oleksii/.python-eggs/MySQL_python-1.2.2-py2.5-macosx-10.3-i386.egg-tmp/_mysql.so, 2): image not found

I finally decided just to look on what actually was in the .python-eggs folder. Rrrriiight:

-rw------- 1 oleksii oleksii 123024 Sep 20 2008 _mysql.so

Isn’t this just beautiful? I don’t get why it is created under my user anyway while Apache2 is running under different user, but since the created library was only readable by me myself… Well, Apache just couldn’t get to it. Damn. What a loss of time (simply not mentioning all previous frustrations with installs, etc).

Reviewing svn changes

I have a habit of reviewing my changes before committing them to the repository. What suprises me is that so far I have only seen home-brewed scripts to perform a ‘batch’ review of all changed files. But ok, how difficult is that? So for my current project I came up with a similar solution. Here are some lines of bash script:


#!/bin/sh

# Find the list of changed files.
FILES=`/usr/local/bin/svn st | grep ^M | /usr/bin/sed ‘s/^M[ ]*\(.*\)/\1/’`
for FILE in $FILES; do
# Invoke favorite diff viewer to see the changes.
svn diff –diff-cmd /Users/oleksii/Scripts/diffwrap.sh $FILE
done

It could have been beautified a bit, but what a heck… BTW, the ‘magic’ diffwrap.sh’ looks like the following for me (use any other diff tool if you like, I just happen to use DiffMerge because it is available for all the platforms I am using now):


#!/bin/sh

DIFF=/Applications/DiffMerge.app/Contents/MacOS/DiffMerge

LEFT=${6}
RIGHT=${7}

$DIFF $LEFT $RIGHT

It should not be difficult to see which OS I am using here :).

XPlanner setup

I like XPlanner, but I am also very suspicious on beta’s being frozen for years (ok, GMail being a nice exception :)). Anyway, today I was happily changing XPlanner configuration to get around a known bug in XPlanner, which fortunately didn’t cost too much time to find solution for. XPlanner on Tomcat 5.5 + Java-6 on Ubuntu 8.04 apparently result in an exception during startup. The following explanation (thanks Alex!) gives a quick fix.

  1. in /xplanner/WEB-INF/classes/sbring-beans.xml
  2. for Find the bean id="metaRepository"
  3. replace
    <map>......</map>
    with

    <property name="repositories">
    <bean class="java.util.HashMap">
    <constructor-arg>
    <map>.....</map>
    </constructor-arg>
    </bean>
    </property>

Arrgggh, I feel I am hacking too much just to get get things working while I already expect them to work out of a box. Is this the way all admins feel?..

Killing time with vsftpd

It seems that I am out of luck for the last 24 hours. I have tried to set up virtual ftp users for my server and seen the number of posts on this topic it should have been a piece of cake. I wish… Some good posts worth having a look though (and reading CAREFULLY!):

Several lessons on this topic:

  1. Make sure you have pam_passwd.so. By default it is NOT present on Ubuntu (at least 8.04). Get it by issuing

    sudo apt-get install libpam-pwdfile
  2. Don’t try to be smarter than you are. The following command (without -m switch!) do the trick with the passwd file:

    sudo htpasswd -c /etc/vsftpd/passwd user_name

Damn, was it worth sleepless 4 hours of headaches?

Getting wireless iface up and running on boot under Ubuntu 8.04

Well, this was driving me crazy for several days. My wlan0 interface was not getting ip address from my router, which was pretty bad considering that machine it supposed to be a server. After all I’ve started getting dreadful
WPA: EAPOL-Key Replay Counter did not increase - dropping packet
messages, which I couldn’t get at all.

Setup

  • Ubuntu 8.04 (server)
  • Linksys WMP54G PCI wireless network card (almost stopped trusting Linksys since this one); driver WinXP 64 bit via ndiswrapper
  • Linksys WRT54GL w/Tomato firmware (yeah, finally got to it and it is so much better than the original;

Issues

  • No IP address after restart on the host; the IP address is provided when doing manual
    sudo ifdown wlan0
    and then
    sudo ifup wlan0

Damn… that’s not how I want to run a headless server that I have planned to put somewhere in the house to get rid of the noise (therefore the wireless to make freely movable).

Tips & Tricks

  • Check if you’re running multiple wpa_supplicant instances by running

    ps ax | grep wpa_supplicant

    If so, stop them by issuing

    wpa_cli terminate

  • The following wpa_supplicant.conf worked for me

    ctrl_interface=/var/run/wpa_supplicant
    ctrl_interface_group=0
    eapol_version=1
    ap_scan=2
    fast_reauth=0

    network={
    pairwise=TKIP
    scan_ssid=1
    proto=WPA
    key_mgmt=WPA-PSK
    ssid="..."
    psk=...
    }

    And here is the reference to it in the /etc/network/interfaces:

    auto lo wlan0
    iface wlan0 inet dhcp
    wpa-driver wext
    wpa-conf /etc/wpa_supplicant.conf

    Finally, there was no luck in obtaining DHCP consistently, I have used the same hack as described in the Ubuntu WIFIDocs, and added the
    ifdown wlan0
    ifup wlan0

    to the /etc/rc.local :(. This works properly since it will be executed when the rest is properly started.

    p.s. This seem to be a known bug #53387, so I am not the only one spending my time for nothing. Welcome to the wonderful world of Linux (you have to set it up and you will be setting it up… forever…).

Guest WinXP VM host share access

I use VmWare Server 1.0.6 for testing purposes (great thing because of snapshots and it is free!), but for a while I could only connect to the guest VM from my host. Somehow my guest Windows XP refused to connect to the host machine shares. After all the following worked:

  • Create a regular user on the host computer.
    e.g. go to Control Panel | User Accounts | Advanced (tab), click on the Advanced button
    And select Users | Right mouse click | New User
    enter new user name and password.
    Make sure the “User must change password at next logon” checkbox is not checked; then check then “Password never expires” checkbox.
  • add a new share to the disk/folder you would like to share
  • give the new user rights to use share (remove Anybody from the list for security reasons)
  • connect from your guest machine using
    net use * \\ /user:<host_name>\<new_user>

    you will be prompted for a password.

Voila!

MySQL TIMESTAMP

Original date: 2008-12-27

Apparently using the TIMESTAMP fields in the MySQL the MySQLdb library in python results in the datetime.datetime() objects. A valid value can be set by providing a valid datetime.datetime instance.