Python, wxPython and internationalization (i18n)

The goal of this document is to describe how to internationalize a Python application that uses wxPython for the User Interface. It uses the attached mki18n.py script.

Contents

Background

I18N under Python and wxPython

wxPython and Python support the gettext system for I18N (internationalization1).

GNU gettext system

Internationalization of software is supported by the GNU Translation Project. Although the project focuses on the goal of translating software user interfaces in as many natural languages as possible and does not impose a tool set, it recommends the use of gettext (and you should use gettext with Python and wxPython).

Idea behind gettext

The general idea behind gettext is that you write your source code in English and all the natural language strings are also written in English. The strings are inserted inside the source code, you do not use string resource identifiers. But all strings that must be translatable are surrounded a small macro call: _(). The gettext system supports a large set of programming languages including C, C++ and Python.

In C or C++ _() is a special macro that calls on gettext function macro is replaced by the preprocessor (in C or C++) by a call to a gettext function that will search for the proper string at run time using the original English string as the key to the language dictionary that is currently active. If the active language is English, no translation is performed, if the active language is something else the translation is performed if a matching string is found. In Python the _() function is either mapped explicitly to gettext.gettext() by your application or installed by the gettext class API.

The dictionaries are compiled files (files with the .mo extension2). Now, to create the .mo files, you first parse all of your source code with the gettext tools and they generate a .pot file (.POT stands for Portable Object Template) which is a simply formatted text file that contains all of the English strings that must be translated. Each English string acts as the message identifier for that string. Below each English string is a spot for the translated version of that string. You copy the .pot file into a .po 3 and give it to a human translator. Then you compile the translated .po file into a .mo file that you place inside one of the LC_MESSAGE directories of your system. These directories are named after the natural language they refer to. The natural languages supported are the ISO 639 language codes. These codes are a set of two-character language codes.

Applications normally place the various .mo files inside language target specific sub-directories of the directory ./local . The following directory tree show a directory tree for English (en), French (fr) and Spanish (es) would look like.

  ./locale/en/LC_MESSAGES
  ./locale/fr/LC_MESSAGES
  ./locale/es/LC_MESSAGES

The .mo files for each language is stored inside the LC_MESSAGES sub-directory under the language code directory. For example, the .mo dictionary file is located inside ./locale/es/LC_MESSAGES.

gettext tools

The GNU gettext tools are available for all OS supported by GNU. The following tools are console tools for the Win32 platform. The GNU tool package include the console programs listed in the following table.

Program

Description

gettext.exe

Displays native language translation of a textual message.

iconv.exe

Decodes text from one encoding to another (e.g. KOI8-R to CP1251)

msgattrib.exe

Filters the messages of a translation catalog according to their attributes, and manipulates the attributes.

msgcat.exe

Concatenates and merges the specified PO files.

msgcmp.exe

Compare two Uniforum style .po files to check that both contain the same set of msgid strings.

msgcomm.exe

Find messages which are common to two or more of the specified PO files.

msgconv.exe

Converts a translation catalog to a different character encoding.

msgen.exe

Creates an English translation catalog. The input file is the last created English PO file, or a PO Template file (generally created by xgettext). Untranslated entries are assigned a translation that is identical to the msgid.

msgexec.exe

Applies a command to all translations of a translation catalog. The COMMAND can be any program that reads a translation from standard input. It is invoked once for each translation. Its output becomes msgexec's output. msgexec's return code is the maximum return code across all invocations.

msgfilter.exe

Applies a filter to all translations of a translation catalog.

msgfmt.exe

Generate binary message catalog from textual translation description.

msggrep.exe

Extracts all messages of a translation catalog that match a given pattern or belong to some given source files.

msginit.exe

Creates a new PO file, initializing the meta information with values from the user's environment.

msgmerge.exe

Merges two Uniforum style .po files together.

msgunfmt.exe

Convert binary message catalog to Uniforum style .po file. msguniq.exe Unifies duplicate translations in a translation catalog.

ngettext.exe

Display native language translation of a textual message whose grammatical form depends on a number.

xgettext.exe

Extract translatable strings from given input files.

How to get gettext tools for Win32

To install the GNU gettext on your Win32 system, follow the instructions:

The files listed above were taken from the following sites:

There are other packages maintained by other individuals. I recommend you use the one above. However, the following sites helped me getting started.

Some cautionary notes:

You should never use a version of gettext older than 0.10.39 (because it produces .po files that cannot be used by the Translation Project without human editing). Version 0.11 is considered stable according to the Translation Project gettext is a set of command line tools and code libraries that have been developed under the GNU umbrella.

gettext file formats

To be written.

Python gettext

The Python gettext module provides internationalization (I18N) and localization (L10N) services for Python modules and applications. It is based on the GNU gettext system.

How to create gettext catalog files

To help creating the gettext binary catalog files, I wrote a Python console program called mki18n.py tha uses the GNU gettext utilities to parse Python source code and create the .po and .mo files. The mki18n.py is used to perform several tasks:

The mki18n.py module can also be used as an imported module inside other Python programs.

How to write an internationalized wxPython application

The application must contain the following code:

   1   gettext.install('ivcm', './locale', unicode=False)

   1   self.presLan_en = gettext.translation("ivcm", "./locale", languages=['en'])
   2   self.presLan_fr = gettext.translation("ivcm", "./locale", languages=['fr'])
   3   self.presLan_es = gettext.translation("ivcm", "./locale", languages=['es'])

   1   self.presLan_fr.install()

   1   self.locale = wx.Locale(wxLANGUAGE_FRENCH)
   2   locale.setlocale(locale.LC_ALL, 'FR')

   1   aTitle = _("Testing internationalization")

The gettext.install(domain, localedir, unicode) call instructs the gettext system to look for the dictionary file name built from the components:

Python and wxPython modules, classes and functions

Controlling the presentation language from the environment variables

The following environment variables control the selection of the translation language. The system uses the language code found in the first environment variable found from the following list:

Tools to manage translation dictionary files

I normally use the CRiSP editor to edit .po files and compare several versions of the .po files. There are, however, specialized tools that simplify managing gettext catalog files. These tools are listed here.

Creating .mo files with the [[attachment:mki18n.py]] script

The mki18n.py script helps you create .po and .mo files from your source code files for an application. I describe the process of internationalizing the ivcm application here.

All strings that must be internationalized inside ivcm.py (and its companion files) have the form _("Hello"). All strings inside the source code are normally written in English as this is the convention used by the gettext system.

To use my mki18n.py script, I write a file called app.fil that contains the names of all files inside the application (one file per line, with full or relative path. For example:

  images.py
  ivcm.py
  ivcm_about.py
  ivcm_ie.py
  ivcm_usermanual.py
  ivcm_wxFrame1.py
  ../ptz.py
  ../action.py
  ../utprint.py

Then I run mki18n -p from the directory where ivcm.py is located to parse all source files and create a 'messages.pot' file. The .pot is the original template. You keep this file untouched. If I want to support French then I copy the messages.pot into a .po file named after the domain name (in this case the application name: 'ivcm') and the target language code (in this case: 'fr'). So for French I use the file name: ivcm_fr.po. If I need to support Spanish, I copy messages.pot into ivcm_es.po and so on.

The following lines show a couple of entries inside the non translated ivcm_fr.po:

  #: ivcm.py:168
  #, python-format
  msgid ""
  "\n"
  "   ERROR: %s"
  msgstr ""

  #: ivcm_wxFrame1.py:638 ivcm_wxFrame1.py:1742
  msgid "&About..."
  msgstr ""

The next step is to perform the translation. You can use a normal editor to append the French string inside the ivcm_fr.po or use poEdit or any other .po editor. The result of the translation would look like:

  #: ivcm.py:168
  #, python-format
  msgid ""
  "\n"
  "   ERROR: %s"
  msgstr ""
  "\n"
  "   ERREUR: %s"

  #: ivcm_wxFrame1.py:638 ivcm_wxFrame1.py:1742
  msgid "&About..."
  msgstr "&A propos de iVCM..."

Note that every line with a '#' in the first column is a comment or flag. In the example above the python-format flag shows that the strings were extracted from Python source. The #: lines show the line number of the original source.

Some of the flags are set when you re-synchronyze the translations with the source. This resynchronization is required if the source changes after you have created the translated .po file(s).

My mki18n.py script will automatically perform syncronisation if it finds .po files that have the domain_language.po name layout. After a re-synchronization, 'mki18n -p' creates a .new file for every .po file found. In my example, it would create a ivcm_fr.po.new and a ivcm_es.po.new

If the source has not changed, the .new files are equal to the .po file. Otherwise, the .new file contains the new strings to translate, place the string that were removed from the source as comments inside the .po.new file and may also flag some strings as 'fuzzy'. A fuzzy flag indicates that the translation of the original source should probably change because the original string changed. So, I compare the .po and .po.new, and edit whatever is requiered, leaving the finished work inside the .po file.

The final step is to compile the finished .po file into the .mo file.

The .mo file normally reside inside the LC_MESSAGES of a 'locale' sub-directory with a xx/LC_MESSAGES for each supported language:

        ./locale
        ./locale/en
        ./locale/en/LC_MESSAGES/ivcm.mo
        ./locale/es
        ./locale/es/LC_MESSAGES/ivcm.mo
        ./locale/es/LC_MESSAGES/wxstd.mo
        ./locale/fr
        ./locale/fr/LC_MESSAGES/ivcm.mo
        ./locale/fr/LC_MESSAGES/wxstd.mo

The ivcm.mo in the fr and sp sub-directories were created mo file by compiling the ivcm_fr.po and ivcm_es.po by using the single command mki18n -m. The ivcm.mo stored inside the en sub-directory was created using the mki18 -e command. The mki18n is a Python scrip that uses the GNU gettext utilities.

The wxstd.mo files contain the compiled string dictionaries for wxPython. These files are distributed with wxPython and are stored under the Python/Lib/site-packages/wxPython/locale/xx/LC_MESSAGES directories (where xx is the language code). Just take them can copy them in your locale directories.

When your application runs, it uses the .mo files identified by your domain (which is ivcm in this case) and the directory (here ./locale) specified by the call gettext.install('ivcm', './locale', unicode=False). The wxPython system uses the wxstd.mo files.

py2exe and gettext

The following gives some sample py2exe setup files which are adapted for gettext .mo files.

Switching between Left to Right and Right to Left Languages

For an application that must be internationalized into languages that read from right to left, such as Hebrew or Arabic, there are other considerations besides translating the strings. The layout of the GUI should be able to switch layout order as well. This is especially important for when options or controls follow a determined direction in the thought process, as having this backwards will confuse the user.

Thankfully, this can be done with rather easily, but the use of sizers is required. You should be using sizers and relative sizes for widgets if you are creating an In8l application anyway, so this shouldn't be a problem.

First we need to determine the language direction. Assume the language variable is a string of the ISO 639-1 code.

   1 right_left_languages = ('ar', 'dv', 'fa', 'ha', 'he', 'ps', 'ur', 'yi')
   2 if language not in right_left_languages:
   3     langLTR = True
   4     alignment = wx.ALIGN_LEFT
   5 else:
   6     langLTR = False
   7     alignment = wx.ALIGN_RIGHT

Now when the elements are set in place at creation time, we can use the langLeftToRight variable to determine the order in which they should be placed in the sizer:

   1 MrSizer = wx.BoxSizer(wx.HORIZONTAL)
   2 # list containing the elements to place
   3 # each element tuple contains:(element, proportion, padding)
   4 SizerElements = [(self.repl_move,0,10),
   5                     (self.repl_move_pos,0,3),
   6                     (self.repl_move_pos_value,0,20),
   7                     (self.repl_move_txt,0,3),
   8                     ((10,10),0,0), # don't forget about spacers!
   9                     (self.repl_move_txt_mod,0,3),
  10                     ((10,10),0,0),
  11                     (self.staticText1,0,3),
  12                     (self.repl_move_txt_value,1,3),
  13                     (self.repl_move_txt_re,0,5),]
  14 # left to right languages
  15 if langLTR:
  16     for i in SizerElements:
  17         MrSizer.Add(i[0],i[1],wx.ALIGN_CENTER|wx.RIGHT,i[2])
  18 # right to left languages:
  19 else:
  20     moveRowElements.reverse()
  21     for i in SizerElements:
  22         MrSizer.Add(i[0],i[1],wx.ALIGN_CENTER|wx.LEFT,i[2])

Sometimes, all that is necessary is changing the alignment, for example when a single element occupies an entire row. In this case the alignment variable suffices.

   1 MrsSizer = wx.BoxSizer(wx.VERTICAL)
   2 MrsSizer.Add(self.staticText2,0,wx.ALL|alignment,5)
   3 MrsSizer.Add(self.staticText3,0,wx.ALL|alignment,5)

Examples

Example 1

Here's an example I made while learning this. I hope it helps someone. -- Nate Silva

   1 """
   2 How to initialize the two translation systems: Python and wxWidgets.
   3 """
   4 import sys, os
   5 import gettext
   6 import wx
   7 # Hack to get the locale directory
   8 basepath = os.path.abspath(os.path.dirname(sys.argv[0]))
   9 localedir = os.path.join(basepath, "locale")
  10 langid = wx.LANGUAGE_DEFAULT    # use OS default; or use LANGUAGE_JAPANESE, etc.
  11 domain = "messages"             # the translation file is messages.mo
  12 # Set locale for wxWidgets
  13 mylocale = wx.Locale(langid)
  14 mylocale.AddCatalogLookupPathPrefix(localedir)
  15 mylocale.AddCatalog(domain)
  16 
  17 # Set up Python's gettext
  18 mytranslation = gettext.translation(domain, localedir,
  19     [mylocale.GetCanonicalName()], fallback = True)
  20 mytranslation.install()
  21 
  22 if __name__ == '__main__':
  23     # use Python's gettext
  24     print _("Hello, World!")
  25 
  26     # use wxWidgets' translation services
  27     print wx.GetTranslation("Hello, World!")
  28 
  29     # if getting unicode errors try something like this:
  30     #print wx.GetTranslation("Hello, World!").encode("utf-8")

Alternately, to just run everything through the wxWidgets translation system, do something like this:

   1 mylocale = wx.Locale(langid)
   2 mylocale.AddCatalogLookupPathPrefix(localedir)
   3 mylocale.AddCatalog(domain)
   4 _ = wx.GetTranslation
   5 #if you are getting unicode errors, try something like:
   6 #_ = lambda s: wx.GetTranslation(s).encode('utf-8')

Example 2

Ianaré Sévi, 2006/07/04

While the two examples above work flawlessly in Windows, when porting over to Mac OS X and Linux I noticed some problems. After much trial and error here is what should be a more complete solution. You will notice I am getting the language info from the file 'language.ini' - all it contains is an internal code for my application depending on what the user has set using another function. In this way language settings 'stick'. I am using the unicode version of wxPython, and this example has been tested successfully on win NT/2000/XP, Linux (Gnome), and Mac OS X86. The little Linux hack is to get things like calendars and stock buttons to display in the proper language.

   1 import wx
   2 import os
   3 import platform
   4 import codecs
   5 import sys
   6 import gettext
   7 def main():
   8     # initialise language settings:
   9     path = sys.path[0].decode(sys.getfilesystemencoding())
  10     try:
  11         langIni = codecs.open(os.path.join(path,u'language.ini'),'r', 'utf-8')
  12     except IOError:
  13         language = u'en' #defaults to english
  14         pass
  15     else:
  16         language = langIni.read()
  17 
  18     locales = {
  19         u'en' : (wx.LANGUAGE_ENGLISH, u'en_US.UTF-8'),
  20         u'es' : (wx.LANGUAGE_SPANISH, u'es_ES.UTF-8'),
  21         u'fr' : (wx.LANGUAGE_FRENCH, u'fr_FR.UTF-8'),
  22         }
  23     mylocale = wx.Locale(locales[language][0], wx.LOCALE_LOAD_DEFAULT)
  24     langdir = os.path.join(path,u'locale')
  25     Lang = gettext.translation(u'messages', langdir, languages=[language])
  26     Lang.install(unicode=1)
  27 
  28     if platform.system() == 'Linux':
  29         try:
  30             # to get some language settings to display properly:
  31             os.environ['LANG'] = locales[language][1]
  32 
  33         except (ValueError, KeyError):
  34             pass
  35 
  36 
  37     #-> Code to launch application goes here. <-#
  38 
  39 
  40 if __name__ == '__main__':
  41     if 'unicode' not in wx.PlatformInfo:
  42         print "\nInstalled version: %s\nYou need a unicode build of wxPython to run this application.\n"%version
  43     else:
  44         main()

- Also, use only ascii characters as your translatable strings (msgid) but use the unicode build of wxPython to display the strings returned by gettext (msgstr). This should allow support for all human languages by your application, however OS must include support as well (ie need to download Windows language pack to correctly view Japanese translation).

Comments

   1 address[::-2].encode('nospam@31tor.com'[7:-4][::-1])


Why you mix gettext and the wx.Locale class, just do it like this:

   1 def installwxgettext():
   2     import wx
   3     import __builtin__
   4     __builtin__._ = wx.GetTranslation

this is the same way like the python gettext module it does, but more elegant i think...


Example 1 gives me a warning " wxStandardPathsBase::Get﴾﴿: create wxApp before calling this" on line containing "mylocale = wx.Locale(langid)" on Win7. I have a chicken and egg problem. I need this set up before making the wx app. At least I think so.

  1. The word internationalization is often abbreviated as I18N: take the first letter of the word (i) followed by the number of letters in the word (18) and terminate it with the last letter of the word (n) (1)

  2. MO stands for Machine Object file. (2)

  3. PO stands for Portable Object file. (3)

  4. The translation domain corresponds to the .mo dictionary file that is searched by the _() gettext translation function. It is selected by gettext.install(). (4)

Internationalization (last edited 2011-09-14 13:54:01 by gw)

NOTE: To edit pages in this wiki you must be a member of the TrustedEditorsGroup.