The purpose of the GUI Unit Testing Project is to create a cross-platform, automated unit testing framework that exercises the GUI nature of wxPython. This is a Google Summer of Code project, submitted by FrankTobia, mentored by Kevin Ollivier.
Justification
Unit testing is a software development best practice designed to ensure code's proper functionality, to avoid bug regressions, to allow for confidence in refactoring, and to improve code quality. Automated unit testing of GUIs is a non-trivial task, with much to be gained from it. The goal of this project is to create an automated, cross-platform GUI unit testing solution for wxPython. wxPython will have a suite of GUI unit tests to ensure correct behavior across multiple platforms. Additionally, it should be easy to develop a suite of unit tests for one's own wxPython program.
Overview
A suite of unit tests is being developed to exercise wxPython's API. It is still alpha-quality, but is stable enough for people to start trying it out. Check out the Unit Testing Quick Start Guide if you're interested in trying it out.
The test suite is effectively a layer of executable documentation. In the ideal case, wxPython's documentation will exactly match the unit test suite, which itself will have all tests passing on all implementations. Of course, this is a long way off. One benefit of the unit test suite is being able to see wxPython code in action, along with the correct output expected. Also, the fact that the documentation is 'executable' means that the implementation can be trivially checked against the documentation to detect discrepancies in either.
It should be noted that in order to write unit tests for a given class, that class's expected behavior must be known. Many classes in wx are either documented incompletely or not at all; in addition to improving the Test Suite, this project aims to improve the documentation, since these activities go hand-in-hand.
Our current effort is being directed towards having all test failures across all platforms reflect meaningful bugs or inconsistencies in wx, rather than bugs or oversights in the test framework. This entails inspecting each test failure, determining (and documenting) the unit's expected behavior through community feedback, and ensuring that the test enforces that expected behavior.
When this is achieved, the wxPython and wxWidgets projects can begin reaping the primary benefit of unit testing; namely, to have a means of limiting bug regressions and ensuring proper functioning of the toolkit. The test suite will be expected to pass on all current versions of wxPython/wxWidgets, and all failures will be indications of bugs that need fixing.
Unit Test Suite Explained
This section is for those interested in knowing the basics of the test suite. For more in-depth ASDF see dev guide. If there's something you'd like to know that isn't listed here, feel free to contact Frank Tobia (contact information below).
The unit test suite supports Python 2.4 and above, so use of 2.5-only features is considered a bug (for now). Also, the unit test suite is currently only being developed for Mac, Windows, and GTK.
Architecture
The suite is built using Python's unittest module. The test suite comprises a test-runner script, a set of test modules, and a utility module.
Each class currently under test in wx has a corresponding module in the test suite. The file containing the test class is named testXxx.py, and the test class is named XxxTest, where Xxx is the name of the wxPython class. There is generally one test class per file. For example, wx.Button has a test class named ButtonTest located in testButton.py. The test-runner script uses these naming conventions to dynamically discover each class in the unit test suite at runtime.
There may be additional utility functions related to the class under test within each file. For example, testSize.py has a function getSizes which returns a list of valid wx.Size instances for the particular class under test.
The suite has an inheritance hierarchy mirroring that of the classes in wx. Generally, if a class in wx has a parent class, its test class will have as its superclass the corresponding parent's test class. For example, wx.Button is derived from wx.Control, which is itself derived from wx.Window; ButtonTest is derived from ControlTest, and ControlTest is derived from WindowTest. This way each test in a base class can be run on its derived classes "for free". Tests can also be overridden or "turned off" for a derived class.
Sometimes a class cannot be instantiated itself: subclassed tests will derive from XxxBase rather than XxxTest. For example, wx.ControlWithItems is an abstract base class. In testControlWithItems.py, ControlWithItemsTest ensures that attempting to instantiate wx.ControlWithItems raises an exception, and ControlWithItemsBase is used for subclassed tests.
Each test class is a subclass of unittest.TestCase.
Cross-platform Reporting
The cross-platform nature of wxPython creates additional challenges for a robust test suite, and at the same time makes one all the more necessary. Since behavior often differs across platforms, the test suite must be run on many different platforms and configurations of wx and wxPython, and this data must be shared. Using the '-w' flag of runAllTests.py, output can be generated in wiki-format with information on the user's platform and configuration. These data can be copied to the Test Suite Results page to be shared with the community.
This feature reduces the legwork of reporting results across platforms. Users with platforms which are not represented in the Test Suite Results are encouraged to run the test suite and upload the results.
Roadmap
Here are some big things we plan on implementing, and the general order in which they'll happen.
- More classes under test
- More tests for classes already under test
- Automatic reporting mechanism
Contact
Anyone with questions, comments, patches, improvements, suggestions, or anything else is encouraged to contact me. Either email the wxPython-dev list with an appropriate subject line, or email me at: frank _dot_ tobia _at_ gmail _dot_ com