So once again I have stumbled upon a script where I needed to send notifications. E-mail comes as the default choice, but there are so many referneces and… what works on one machine does not seem to work on another. Why?.. Read on.
Tag: python
Python notes
This post is inspired by post (in Russian) on
Habrahabr. Few things were not that apparent for already a few years of using Python, but can be very handy!
Continue reading “Python notes”
Configuring Django under wsgi
A small reminder for settings up Django under wsgi. Apparently the following snippet helps getting the path correct avoiding the dreadful
self.load_middleware()
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/core/handlers/base.py", line 39, in load_middleware
for middleware_path in settings.MIDDLEWARE_CLASSES:
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/utils/functional.py", line 184, in inner
self._setup()
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/conf/__init__.py", line 95, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'xxx.settings' (Is it on sys.path?): No module named xxx.settings
messages from Apache (well, instead of ‘xxx’ your project name will be mentioned of course).
The fix
import sys
# Correct path.
app_path = os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..'))
if app_path not in sys.path:
sys.path.append(app_path)
For completeness, the sites-enables/xxx looks like
WSGIPythonPath /var/www/path/to/project
<VirtualHost *:80>
DocumentRoot /var/www/path/to/project
ServerAdmin admin@example.com
ServerName my.example.com
Alias /static/ /var/www/path/to/project/
Order deny,allow
Allow from all
WSGIDaemonProcess project_name
WSGIScriptAlias / /var/www/path/to/project/wsgi.py
</VirtualHost>
Happy Djangoing!
UPDATE: when multiple Django applications are configured under Apache apparently a request could be routed to the Apache instance hosting wrong application. The server will not load application and Apache log files would show:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/core/handlers/wsgi.py", line 219, in __call__
self.load_middleware()
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/core/handlers/base.py", line 39, in load_middleware
for middleware_path in settings.MIDDLEWARE_CLASSES:
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/utils/functional.py", line 184, in inner
self._setup()
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/conf/__init__.py", line 95, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'xxx.settings' (Is it on sys.path?): No module named xxx.settings
basically failing to load settings from the wrong application. More on this topic in this blog post, but the fix seem to be specifying WSGIDaemonProcess per application.
Setting up Dango i18n, i10n
Anything related to i18n/i10n subject seem to be somehow quirkier than it looks at first glance. Python (2.x) itself and handling of unicode is a story apart.
This time I was looking into building a small website that has to provide UI in different languages. As this is one of the things you want to have right away I’ve started experimenting with adding i18n and i10n support.
First step is easy, the settings.py already had proper settings. Then for .py files it is rather straightforward to add e.g. (for forms):
from django.utils.translation import ugettext_lazy as _
...
city_name = forms.CharField( required = False, label = _('City:'))
For the .html files something like
{% load i18n %}
{% trans "Hello there!" %}
Then create under project folder folder ‘conf/locale’ (if you don’t do this it will complain), and then run
django-admin.py makemessages -l ru
Edit the resulting django.po file, add translations to your messages.
Warning: don’t forget to edit the following field, which comes EMPTY first, even while you have given it a parameter! Otherwise this file will be not used properly.
"Language: ru\n"
Then compile your nice and shiny translations:
django-admin.py compilemessages
Now we get all messages available. At least they should. But there is another trick missed in the Django documentation/tutorials: you HAVE TO specify the location of the message files explicitly in your settings.py otherwise your texts will continue coming up in default (en) language no matter how hard you try. E.g.:
LOCALE_PATHS = (
os.path.join(os.path.dirname(__file__), 'conf', 'locale').replace('\\','/'),
)
Well, after all this it seems to work. But it costs quite some searching and poking around to come to this. I can imagine after several rounds this becomes obvious, but you don’t get any errors, warnings, whatsoever, it just does not what you want it to do. Well, I hope it will do it for you now :).
Happy Djangoing!
A catch with the clean_xxx methods in Django forms
As I am getting more into Django stuff I also seem to step into rookie mistakes. Hereby some (beginner) notes.
As I am getting more into Django stuff I also seem to step into rookie mistakes. Hereby some (beginner) notes.
For validating of a particular field in Django a form class method can be defined, e.g.
from django import forms
class MyForm(forms.Form):
first_field = forms.CharField()
second_field = forms.CharField()
def clean_first_field(self):
first_field = self.cleaned_data['first_field']
Well, so far so good. But there is a catch which cost me another sleepless hour last night. When the method for field1 is called the ‘self.cleaned_data’ dictionary does not contain the ‘field2’ yet!!
Try the following:
def clean_field1(self):
first_field = self.cleaned_data['first_field']
second_field = self.cleaned_data['second_field'] # This will result in a KeyError exception!
Apparently Django will call the clean_xxx() methods in the same sequence as they are defined and will only supply the values UP TO AND INCLUDING the field being ‘cleaned’, but NOT the fields defined AFTER it. Watta…
This means if you want to check e.g. field1 and field2 between each other (yes, there are other ways of doing that as well, but when you’re only so far through the Django book you don’t know about them :)), then you have to check for the LAST field you want to check against to get values of this and PREVIOUSLY defined fields. So the code becomes:
def clean_first_field(self):
first_field = self.cleaned_data['first_field'] # This is OK, field1 is defined before field2
second_field = self.cleaned_data['second_field'] # As this is called for field2 you can get its value as well.
Of course you should have more error-checking logic, the examples above are simplified for illustrative reasons.
Note that if you use the higher-level clean() method it will be called when all values are filled in.
More on MySQLdb on Snow Leopard
Well, my frustrations with MySQL and MacOSX continue. This time after a copmlete clean reinstall of everything I have stumbled upon the ‘old friend’ MySQLdb Python module. Apparently setting up this one causes a lot of frustration and blogging. Here is what I have found.
1. Check your platform. I have an old MacBook and naively installed 32 bit MySQL on it. Man, I was wrong. Since I wanted to use the default MacOSX Python I should have checked that one fist. You can do that by executing the following commands in the Python interactive shell:
import sys
import math
math.log(sys.maxint, 2)
It gave me… “63.0”. Wow… Apparently I do have 64-bit Python!
2. Having 64 bit Python you must install 64 bit MySQL. So go and get the proper version.
3. If you think you’re there you’re not getting my frustration yet. If you would get MySQLdb, build it now and run as I did, the following statement
import MySQLdb
will likely to result in
Traceback (most recent call last):
File "
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/oleksii/.python-eggs/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Library not loaded: libmysqlclient.18.dylib
Referenced from: /Users/oleksii/.python-eggs/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Reason: image not found
Ok, this was it. I had to dig around and finally found out that it is a known bug in the at least 5.5 version of MySQL, explanation and the fix can be looked up at the MySQL bugs website.
In short you have to perform the following
sudo install_name_tool -id /usr/local/mysql/lib/libmysqlclient.18.dylib libmysqlclient.dylib
where ’18’ needs to be replaced with your version.
4. Finally get the MySQLdb (I’ve got 1.2.3). Build it, make sure the proper architecture is used. The architecture can be enforced with e.g. the following commands:
export VERSIONER_PYTHON_PREFER_64_BIT=yes
(or the coresponding one for 32 bits)
Wow, finally import worked as I had expected. WTFx10…
…and all this after several hours debugging for a problem that existed for 10+ years and appeared due to changes in the new Visual Studio compiler optimizations… great… another sleepless night…
Remove non-ascii characters in file names
Today someone asked to help with getting some files with non-ASCII characters on his Linux box. The problem was that those file couldn’t be read by some apps (unicode remains a mystery for some). Since I am not a bash-minded person I thought of Python…
Today someone asked to help with getting some files with non-ASCII characters on his Linux box. The problem was that those file couldn’t be read by some apps (unicode still remains a mystery for some). Since I am not a bash-minded person I thought of Python (2.x) first (works on Windows as well). The following script will remove any non-ASCII characters from file names.
WARNING: Beware if you have files that may end up in the same name, as the files may be overridden!
import os
for file in os.listdir(u"."):
if os.path.isfile(file) and file.endswith(u'.rar'):
new_file = "".join(i for i in file if ord(i)<128)
if (file != new_file):
print u"Renaming", file.encode('utf8'),u" to ", new_file.encode('utf8')
os.rename(file, new_file)
Note that the u"." is essential so that you get the unicode file names back. The "." will give you regular string which is pain-in-the-butt. Sticking to bash (if you don’t like Python for some reason), I’ve came up with the following script:
for f in *.rar; do
mv "$f" `echo $f | tr -cd "a-zA-Z0-9.-_"`
done
Note that most script deal with data IN the files, but not the file names themselves. I hope some other people can use this as well.
Python.NET and VS2010/.NET 4
I hope this will help some people like me searching for an answer. Several steps were mentioned in the Python for .NET mailing list by other people as well, but I haven’t seen a step-by-step guide. It is not my intention to duplicate other posts in that sense, but rather have all-in-one post.
Here is how I’ve got VS2010 and .NET 4.0 working with the revision 122 of the Python.NET having Python 2.6 installed.
Compile
- Get the sources (tarball from sourceforge or directly from SVN)
- Open the pythonnet.sln with VS2010 and convert to 2010 format (will happen automagically)
- [updated] Add the constructorbinding.cs to the Python.Runtime.csproj project (see also this post in the PythonNET mailing list)
- Change the target framework to 4. Follow the following step for EACH project
Right-click on the project name and select “Properties”
Select the “Application” tab on the left (if not selected yet)
Change the “Target framework” to “.NET Framework 4” - Open the buildclrmodule.bat and change the following lines (2 times!)
%windir%\Microsoft.NET\Framework\v2.0.50727\ilasm /nologo /quiet /dll %ILASM_EXTRA_ARGS% /include=%INCLUDE_PATH% /output=%OUTPUT_PATH% %INPUT_PATH%
to
%windir%\Microsoft.NET\Framework\v4.0.30319\ilasm /nologo /quiet /dll %ILASM_EXTRA_ARGS% /include=%INCLUDE_PATH% /output=%OUTPUT_PATH% %INPUT_PATH%
- Open the clrmodule.il and change the lines with the version number in the following piece of code.assembly extern mscorlib
{
.publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
.ver 2:0:0:0
}
to
.ver 4:0:0:0
- Recompile the whole solution, ignore the deprecation warnings.
Now you have all necessary files under the pythonnet folder where you have the sources. You need clr.pyd, python.exe and Python.Runtime.dll.
Test
- Run the newly compiled python.exe
Type the following in the interactive prompt>>> import System
>>> print System.Environment.Version
4.0.30319.1The last line proves that you’re using the 4.0 runtime. The precompiled binaries available from SourceForge would show
2.0.50727.3615
python xmlrpclib “cannot marshal None unless allow_none is enabled”
Just another puzzle before the weekend. After some refactoring and tearing things apart to make them testable (funny enough after a 2 day TDD course) I have stumbled upon the following error (replaced the methods/parameters with stars…):
http://localhost:8051.
Traceback:
File "*****.py", line 242, in run *****(*****)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1147, in __call__ return self.__send(self.__name, args)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1437, in __request verbose=self.__verbose
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1201, in request return self._parse_response(h.getfile(), sock)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response return u.close()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 787, in close raise Fault(**self._stack[0])
Apparently while putting things into separate methods I have omitted return values to be properly returned from the inner to the outer methods, which in Python world means None return value. So the client from an XMLRPC client to the server returned None, which is apparently not handled by the standard XMLRPC protocol (unless some extensions like suggested ‘allow_none’ are used).
So again, if it worked, but then ‘suddenly’ stopped, check if the method called via XMLRPC is properly ending with a return statement which is not evaluating to None.
Working with unicode in Python (again)
This time I have stumbled (again) a unicode problem using some Python code which was supposed to be perfectly suitable for doing this since it even started with
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
It was quite some time since the last post, but this does not mean I haven’t done anything interesting :). It is just that it was so much interesting that I didn’t have any time to write anything.
Anyway, this time I have stumbled (again) a unicode problem using some Python code which was supposed to be perfectly suitable for doing this since it even started with
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
It went perfectly fine when running in Eclipse, but to my huge surprise I’ve got problems when running the unit tests from the command line in terminal. Whaaat? It just worked!
Well, declaring your source as UTF-8 is not enough of course. There are several things to check when getting the “UnicodeDecodeError: ‘ascii’ codec can’t decode byte … in position …: ordinal not in range(128)”-kind of errors. Googling around didn’t bring me much luck to my surprise, so there is are my findings for the next time :).
First of all make absolutely sure you haven’t forgotten the ‘u’ character before your strings containing the unicode strings. Yep, just like that you screw up the rest of the unicode support. Python (ok, I admit, I use 2.5.4) treats a ‘string’ as a regular string and not as a unicode. So write u’string’ instead!
Second, when doing things file operations don’t forget that you don’t get the unicode by default. Consider the following:
message = u'unicode message'
file_handle.write(message)
Well, guess what. You get a problem when writing the string away. It cannot be recognized. So the solution would be to do something like this
encoded_message = message.encode(u"utf-8")
file_handle.write(encoded_message)
But that’s only a half of the problem. At some point you will be reading this data back. And most probably you would like to get your beloved unicode thingy back. Just doing the following will hardly help:
file_handle = open(full_name, 'r')
line = file_handle.readline()
The following will save your day:
file_handle = open(full_name, 'r')
line = file_handle.readline().decode('utf-8')
Voila. I hope this saves some frustration to somebody, even if it will be me some month later :).
Enjoy!
Comments from Andy (thanks!):
Probably there is nothing new for you in what I am saying below, however, from my experience, it covers 99% unicode-related errors.
Unicode string is s sequence of code points in range 0 to 0x10ffff. Encoding is a way of serializing this sequence, so thay can be represented in memory, written to a file, sent over a socket etc.
Encoding unicode string is needed _at least_ because its ‘as is’ byte representation is not portable due to byte order issues.
It is _recommended_ that you work with unicode string internally provided the language/API supports unicode
It is _must_ that you encode the string to be consumed by another program.