Python, wxPython and internationalization (i18n)

The goal of this document is to describe how to internationalize a Python application that uses wxPython for the User Interface. It uses the attached attachment:mki18n.py script.

Contents TableOfContents

Background

I18N under Python and wxPython

wxPython and Python support the gettext system for I18N (internationalizationFootNote(The word internationalization is often abbreviated as I18N: take the first letter of the word (i) followed by the number of letters in the word (18) and terminate it with the last letter of the word (n))).

GNU gettext system

Internationalization of software is supported by the GNU [http://www.iro.umontreal.ca/contrib/po/HTML/index.html Translation Project]. Although the project focuses on the goal of translating software user interfaces in as many natural languages as possible and does not impose a tool set, it recommends the use of gettext (and you should use gettext with Python and wxPython).

Idea behind gettext

The general idea behind gettext is that you write your source code in English and all the natural language strings are also written in English. The strings are inserted inside the source code, you do not use string resource identifiers. But all strings that must be translatable are surrounded a small macro call: _(). The gettext system supports a large set of programming languages including C, C++ and Python.

In C or C++ _() is a special macro that calls on gettext function macro is replaced by the preprocessor (in C or C++) by a call to a gettext function that will search for the proper string at run time using the original English string as the key to the language dictionary that is currently active. If the active language is English, no translation is performed, if the active language is something else the translation is performed if a matching string is found. In Python the _() function is either mapped explicitly to gettext.gettext() by your application or installed by the gettext class API.

The dictionaries are compiled files (files with the .mo extensionFootNote(MO stands for Machine Object file.)). Now, to create the .mo files, you first parse all of your source code with the gettext tools and they generate a .pot file (.POT stands for Portable Object Template) which is a simply formatted text file that contains all of the English strings that must be translated. Each English string acts as the message identifier for that string. Below each English string is a spot for the translated version of that string. You copy the .pot file into a .po FootNote(PO stands for Portable Object file.) and give it to a human translator. Then you compile the translated .po file into a .mo file that you place inside one of the LC_MESSAGE directories of your system. These directories are named after the natural language they refer to. The natural languages supported are the [http://www.gnu.org/manual/gettext/html_chapter/gettext_15.html#SEC221 ISO 639 language codes]. These codes are a set of two-character language codes.

Applications normally place the various .mo files inside language target specific sub-directories of the directory ./local . The following directory tree show a directory tree for English (en), French (fr) and Spanish (es) would look like.

  ./locale/en/LC_MESSAGES
  ./locale/fr/LC_MESSAGES
  ./locale/es/LC_MESSAGES

The .mo files for each language is stored inside the LC_MESSAGES sub-directory under the language code directory. For example, the .mo dictionary file is located inside ./locale/es/LC_MESSAGES.

gettext tools

The GNU gettext tools are available for all OS supported by GNU. The following tools are console tools for the Win32 platform. The GNU tool package include the console programs listed in the following table.

How to get gettext tools for Win32

To install the GNU gettext on your Win32 system, follow the instructions:

• download the following files:

• [ftp://ftp.gnu.org/pub/gnu/gettext/gettext-runtime-0.13.1.bin.woe32.zip gettext-runtime-0.13.1.bin.woe32.zip]

• [ftp://ftp.gnu.org/pub/gnu/gettext/gettext-tools-0.13.1.bin.woe32.zip gettext-tools-0.13.1.bin.woe32.zip]

• [http://mirrors.ibiblio.org/pub/mirrors/gnu/ftp/pub/gnu/libiconv/libiconv-1.9.1.bin.woe32.zip libiconv-1.9.1.bin.woe32.zip]

• Create a directory called \gnu in one of your system drive (let's say in D:)
• Extract the 3 ZIP files inside the \gnu directory. This way the executable files of the 3 packages will be stored inside \gnu\bin.
• Add \gnu\bin to your PATH.

The files listed above were taken from the following sites:

• [ftp://ftp.gnu.org/gnu/gettext/ GNU FTP site for gettetx] where several versions (0.10.40, 0.11.2, 0.11.5 and 0.12.1) are available.

• The [http://www.ibiblio.org/pub/gnu/libiconv/ GNU libiconv ftp site]. This is the ftp site for the [http://www.gnu.org/software/libiconv/ GNU libiconv] library. The iconv library is required by gettext. Get version 1.9.1 or later.

There are other packages maintained by other individuals. I recommend you use the one above. However, the following sites helped me getting started.

• [http://gettext.sourceforge.net/ SourceForge page of gettext for Win32]

• [http://home.a-city.de/franco.bez/gettext/gettext_win32_en.html GNU gettext for WIN32] is a little distribution of the GNU gettext for Win32

Some cautionary notes:

You should never use a version of gettext older than 0.10.39 (because it produces .po files that cannot be used by the Translation Project without human editing). Version 0.11 is considered stable according to the [http://www.iro.umontreal.ca/contrib/po/HTML/index.html Translation Project] gettext is a set of command line tools and code libraries that have been developed under the [http://www.gnu.org GNU] umbrella.

gettext file formats

To be written.

Python gettext

The [http://www.python.org/doc/current/lib/module-gettext.html Python gettext module] provides internationalization (I18N) and localization (L10N) services for Python modules and applications. It is based on the GNU gettext system.

How to create gettext catalog files

To help creating the gettext binary catalog files, I wrote a Python console program called attachment:mki18n.py tha uses the GNU gettext utilities to parse Python source code and create the .po and .mo files. The attachment:mki18n.py is used to perform several tasks:

• parse all Python source code files of an application and create the .pot file for each target language.
• parse all Python source code files of an application and merge the generated .pot with existing .po (files that already contain text translated by human translator).
• create the binary .mo files from the .po files.

The attachment:mki18n.py module can also be used as an imported module inside other Python programs.

How to write an internationalized wxPython application

The application must contain the following code:

• the main module must import the gettext module.

• the global code of the main module must call gettext.install() to install the function _() inside the application dictionary. This call identifies the translation domainFootNote(The translation domain corresponds to the .mo dictionary file that is searched by the _() gettext translation function. It is selected by gettext.install(). ) (which normally is the application name), the location of the LC_MESSAGES directories and whether Unicode is used. The following is an example of the call where the dictionary files are under ./locale/xx/LC_MESSAGES (with xx being the language code) and ivcm being the name of the application (and the domain, and the name of the .mo file):

   1   gettext.install('ivcm', './locale', unicode=False)

• For each supported presentation language, the program must create a gettext. Translation instance by calling gettext.translation(). The arguments to the translation function are the application domain (the name of the .mo file which is often the name of the application), the directory parent of the languages LC_MESSAGES directories and the list of language. For example, if your application is called ivcm and the files are stored under a directory called locales, the support for English, French and Spanish would be set up by doing the following calls:

   1   self.presLan_en = gettext.translation("ivcm", "./locale", languages=['en'])
   2   self.presLan_fr = gettext.translation("ivcm", "./locale", languages=['fr'])
   3   self.presLan_es = gettext.translation("ivcm", "./locale", languages=['es'])

• To activate a the translation of your application for a specified language, the language must be activated by calling the Translation.install() method. For example, to activate the French presentation, you would call:

   1   self.presLan_fr.install()

• Set the wxWindows locale by calling wxLocale() and passing the wxLANGUAGE_XX code corresponding to the selected language. The following code snippet shows you how:

   1   self.locale = wx.Locale(wxLANGUAGE_FRENCH)
   2   locale.setlocale(locale.LC_ALL, 'FR')

• In your code, all strings must be enclosed in a _() call like this:

   1   aTitle = _("Testing internationalization")

The gettext.install(domain, localedir, unicode) call instructs the gettext system to look for the dictionary file name built from the components:

• localedir/language/LC_MESSAGES/domain.mo

Python and wxPython modules, classes and functions

• Python gettext

• wxPython wx.Locale class

• wxPython wx.GetTranslation()

• wxWidgets wxLocale

Controlling the presentation language from the environment variables

The following environment variables control the selection of the translation language. The system uses the language code found in the first environment variable found from the following list:

• LANGUAGE
• LC_ALL
• LC_MESSAGES
• LANG

Tools to manage translation dictionary files

I normally use the CRiSP editor to edit .po files and compare several versions of the .po files. There are, however, specialized tools that simplify managing gettext catalog files. These tools are listed here.

• ["poEdit"] is a cross-platform gettext catalogs (.po files) editor.

Creating .mo files with the attachment:mki18n.py script

The attachment:mki18n.py script helps you create .po and .mo files from your source code files for an application. I describe the process of internationalizing the ivcm application here.

All strings that must be internationalized inside ivcm.py (and its companion files) have the form _("Hello"). All strings inside the source code are normally written in English as this is the convention used by the gettext system.

To use my attachment:mki18n.py script, I write a file called app.fil that contains the names of all files inside the application (one file per line, with full or relative path. For example:

  images.py
  ivcm.py
  ivcm_about.py
  ivcm_ie.py
  ivcm_usermanual.py
  ivcm_wxFrame1.py
  ../ptz.py
  ../action.py
  ../utprint.py

Then I run mki18n -p from the directory where ivcm.py is located to parse all source files and create a 'messages.pot' file. The .pot is the original template. You keep this file untouched. If I want to support French then I copy the messages.pot into a .po file named after the domain name (in this case the application name: 'ivcm') and the target language code (in this case: 'fr'). So for French I use the file name: ivcm_fr.po. If I need to support Spanish, I copy messages.pot into ivcm_es.po and so on.

The following lines show a couple of entries inside the non translated ivcm_fr.po:

  #: ivcm.py:168
  #, python-format
  msgid ""
  "\n"
  "   ERROR: %s"
  msgstr ""

  #: ivcm_wxFrame1.py:638 ivcm_wxFrame1.py:1742
  msgid "&About..."
  msgstr ""

The next step is to perform the translation. You can use a normal editor to append the French string inside the ivcm_fr.po or use poEdit or any other .po editor. The result of the translation would look like:

  #: ivcm.py:168
  #, python-format
  msgid ""
  "\n"
  "   ERROR: %s"
  msgstr ""
  "\n"
  "   ERREUR: %s"

  #: ivcm_wxFrame1.py:638 ivcm_wxFrame1.py:1742
  msgid "&About..."
  msgstr "&A propos de iVCM..."

Note that every line with a '#' in the first column is a comment or flag. In the example above the python-format flag shows that the strings were extracted from Python source. The #: lines show the line number of the original source.

Some of the flags are set when you re-synchronyze the translations with the source. This resynchronization is required if the source changes after you have created the translated .po file(s).

My attachment:mki18n.py script will automatically perform syncronisation if it finds .po files that have the domain_language.po name layout. After a re-synchronization, 'mki18n -p' creates a .new file for every .po file found. In my example, it would create a ivcm_fr.po.new and a ivcm_es.po.new

If the source has not changed, the .new files are equal to the .po file. Otherwise, the .new file contains the new strings to translate, place the string that were removed from the source as comments inside the .po.new file and may also flag some strings as 'fuzzy'. A fuzzy flag indicates that the translation of the original source should probably change because the original string changed. So, I compare the .po and .po.new, and edit whatever is requiered, leaving the finished work inside the .po file.

The final step is to compile the finished .po file into the .mo file.

The .mo file normally reside inside the LC_MESSAGES of a 'locale' sub-directory with a xx/LC_MESSAGES for each supported language:

        ./locale
        ./locale/en
        ./locale/en/LC_MESSAGES/ivcm.mo
        ./locale/es
        ./locale/es/LC_MESSAGES/ivcm.mo
        ./locale/es/LC_MESSAGES/wxstd.mo
        ./locale/fr
        ./locale/fr/LC_MESSAGES/ivcm.mo
        ./locale/fr/LC_MESSAGES/wxstd.mo

The ivcm.mo in the fr and sp sub-directories were created mo file by compiling the ivcm_fr.po and ivcm_es.po by using the single command mki18n -m. The ivcm.mo stored inside the en sub-directory was created using the mki18 -e command. The mki18n is a Python scrip that uses the GNU gettext utilities.

The wxstd.mo files contain the compiled string dictionaries for wxPython. These files are distributed with wxPython and are stored under the Python/Lib/site-packages/wxPython/locale/xx/LC_MESSAGES directories (where xx is the language code). Just take them can copy them in your locale directories.

When your application runs, it uses the .mo files identified by your domain (which is ivcm in this case) and the directory (here ./locale) specified by the call gettext.install('ivcm', './locale', unicode=False). The wxPython system uses the wxstd.mo files.

py2exe and gettext

The following gives some sample py2exe setup files which are adapted for gettext .mo files.

• ["py2exeAndGettext"] what needs to be done to handle .mo files with py2exe

Switching between Left to Right and Right to Left Languages

For an application that must be internationalized into languages that read from right to left, such as Hebrew or Arabic, there are other considerations besides translating the strings. The layout of the GUI should be able to switch layout order as well. This is especially important for when options or controls follow a determined direction in the thought process, as having this backwards will confuse the user.

Thankfully, this can be done with rather easily, but the use of sizers is required. You should be using sizers and relative sizes for widgets if you are creating an In8l application anyway, so this shouldn't be a problem.

First we need to determine the language direction. Assume the language variable is a string of the ISO 639-1 code.

   1 right_left_languages = ('ar', 'dv', 'fa', 'ha', 'he', 'ps', 'ur', 'yi')
   2 if language not in right_left_languages:
   3     langLTR = True
   4     alignment = wx.ALIGN_LEFT
   5 else:
   6     langLTR = False
   7     alignment = wx.ALIGN_RIGHT

Now when the elements are set in place at creation time, we can use the langLeftToRight variable to determine the order in which they should be placed in the sizer:

   1 MrSizer = wx.BoxSizer(wx.HORIZONTAL)
   2 # list containing the elements to place
   3 # each element tuple contains:(element, proportion, padding)
   4 SizerElements = [(self.repl_move,0,10),
   5                     (self.repl_move_pos,0,3),
   6                     (self.repl_move_pos_value,0,20),
   7                     (self.repl_move_txt,0,3),
   8                     ((10,10),0,0), # don't forget about spacers!
   9                     (self.repl_move_txt_mod,0,3),
  10                     ((10,10),0,0),
  11                     (self.staticText1,0,3),
  12                     (self.repl_move_txt_value,1,3),
  13                     (self.repl_move_txt_re,0,5),]
  14 # left to right languages
  15 if langLTR:
  16     for i in SizerElements:
  17         MrSizer.Add(i[0],i[1],wx.ALIGN_CENTER|wx.RIGHT,i[2])
  18 # right to left languages:
  19 else:
  20     moveRowElements.reverse()
  21     for i in SizerElements:
  22         MrSizer.Add(i[0],i[1],wx.ALIGN_CENTER|wx.LEFT,i[2])

Sometimes, all that is necessary is changing the alignment, for example when a single element occupies an entire row. In this case the alignment variable suffices.

   1 MrsSizer = wx.BoxSizer(wx.VERTICAL)
   2 MrsSizer.Add(self.staticText2,0,wx.ALL|alignment,5)
   3 MrsSizer.Add(self.staticText3,0,wx.ALL|alignment,5)

Examples

Example 1

Here's an example I made while learning this. I hope it helps someone. -- Nate Silva

   1 """
   2 How to initialize the two translation systems: Python and wxWidgets.
   3 """
   4 import sys, os
   5 import gettext
   6 import wx
   7 # Hack to get the locale directory
   8 basepath = os.path.abspath(os.path.dirname(sys.argv[0]))
   9 localedir = os.path.join(basepath, "locale")
  10 langid = wx.LANGUAGE_DEFAULT    # use OS default; or use LANGUAGE_JAPANESE, etc.
  11 domain = "messages"             # the translation file is messages.mo
  12 # Set locale for wxWidgets
  13 mylocale = wx.Locale(langid)
  14 mylocale.AddCatalogLookupPathPrefix(localedir)
  15 mylocale.AddCatalog(domain)
  16 
  17 # Set up Python's gettext
  18 mytranslation = gettext.translation(domain, localedir,
  19     [mylocale.GetCanonicalName()], fallback = True)
  20 mytranslation.install()
  21 
  22 if __name__ == '__main__':
  23     # use Python's gettext
  24     print _("Hello, World!")
  25 
  26     # use wxWidgets' translation services
  27     print wx.GetTranslation("Hello, World!")
  28 
  29     # if getting unicode errors try something like this:
  30     #print wx.GetTranslation("Hello, World!").encode("utf-8")

Alternately, to just run everything through the wxWidgets translation system, do something like this:

   1 mylocale = wx.Locale(langid)
   2 mylocale.AddCatalogLookupPathPrefix(localedir)
   3 mylocale.AddCatalog(domain)
   4 _ = wx.GetTranslation
   5 #if you are getting unicode errors, try something like:
   6 #_ = lambda s: wx.GetTranslation(s).encode('utf-8')

Example 2

Ianaré Sévi, 2006/07/04

While the two examples above work flawlessly in Windows, when porting over to Mac OS X and Linux I noticed some problems. After much trial and error here is what should be a more complete solution. You will notice I am getting the language info from the file 'language.ini' - all it contains is an internal code for my application depending on what the user has set using another function. In this way language settings 'stick'. I am using the unicode version of wxPython, and this example has been tested successfully on win NT/2000/XP, Linux (Gnome), and Mac OS X86. The little Linux hack is to get things like calendars and stock buttons to display in the proper language.

   1 import wx
   2 import os
   3 import platform
   4 import codecs
   5 import sys
   6 import gettext
   7 def main():
   8     # initialise language settings:
   9     path = sys.path[0].decode(sys.getfilesystemencoding())
  10     try:
  11         langIni = codecs.open(os.path.join(path,u'language.ini'),'r', 'utf-8')
  12     except IOError:
  13         language = u'en' #defaults to english
  14         pass
  15     else:
  16         language = langIni.read()
  17 
  18     locales = {
  19         u'en' : (wx.LANGUAGE_ENGLISH, u'en_US.UTF-8'),
  20         u'es' : (wx.LANGUAGE_SPANISH, u'es_ES.UTF-8'),
  21         u'fr' : (wx.LANGUAGE_FRENCH, u'fr_FR.UTF-8'),
  22         }
  23     mylocale = wx.Locale(locales[language][0], wx.LOCALE_LOAD_DEFAULT)
  24     langdir = os.path.join(path,u'locale')
  25     Lang = gettext.translation(u'messages', langdir, languages=[language])
  26     Lang.install(unicode=1)
  27 
  28     if platform.system() == 'Linux':
  29         try:
  30             # to get some language settings to display properly:
  31             os.environ['LANG'] = locales[language][1]
  32 
  33         except (ValueError, KeyError):
  34             pass
  35 
  36 
  37     #-> Code to launch application goes here. <-#
  38 
  39 
  40 if __name__ == '__main__':
  41     if 'unicode' not in wx.PlatformInfo:
  42         print "\nInstalled version: %s\nYou need a unicode build of wxPython to run this application.\n"%version
  43     else:
  44         main()

- Also, use only ascii characters as your translatable strings (msgid) but use the unicode build of wxPython to display the strings returned by gettext (msgstr). This should allow support for all human languages by your application, however OS must include support as well (ie need to download Windows language pack to correctly view Japanese translation).

Comments

• Please feel free to provide any type of feedback on this page or on mki18n.py to me. I will be revamping mki18n.py soon. /Pierre Rouleau.
• You can email me at the address provided. I had so much spam at some point, so my email is encoded below.

• The encoded address is: 'zNbOpS.PfAxMe@bMjEgSrAaMuIgGnOcSz vA@PhInPrYyThHbOeNc'

• Use the following code to decode it:

   1 address[::-2].encode('nospam@31tor.com'[7:-4][::-1])

Why you mix gettext and the wx.Locale class, just do it like this:

   1 def installwxgettext():
   2     import wx
   3     import __builtin__
   4     __builtin__.__dict__['_'] = wx.GetTranslation

this is the same way like the python gettext module it does, but more elegant i think...