Willie Walker's Weblog

One of the problematic things with testing any end-user application is the need to have someone (or something) sit behind the application and actually run through it using a manual testing procedure. These tests can often take a mind-numbing number of days for a tester to complete. Thus, given the time and resources involved, the testing may not always be done as frequently as it is needed.

To provide some level for testing in Orca, Orca uses features of the AT-SPI to allow an assistive technology client to record and playback keystrokes. It does so via two very simple (and unsupported) utilities:

• record_keystrokes.py: run this via "python record_keystrokes.py" and Orca will spit out all keystrokes entered by the user, along with timing information. Save this output to a file and you can play the keystrokes using play_keystrokes.py. This application will automatically quit when the user presses "F12".
• play_keystrokes.py: give this a file created from the output of record_keystrokes.py, and it will synthesize keystrokes respecting the timing information in the file.

Couple these two utilities with the debug/logging capabilities of Orca (to be blogged about soon), and one can start generating tests. For example:

• Start the desktop in a well-known state: all empty except for a single terminal window with focus.
• Start Orca with logging information turned on such that Orca says what it is sending to the speech synthesizer and the braille display. Record this logging information to a file.
• From a separate machine, run record_keystrokes, saving the output to a file.
• In the terminal window, type "gedit" or the name of any application you want to test.
• Interact with the application (i.e., "gedit") using the keyboard, performing a variety of tasks.
• Quit the application and press F12 to quit record_keystrokes.

At this point, you now have a file containing keystrokes that can be played back. Due to some various vagaries in the AT-SPI, which should be solved when the XEvIE extension is available on more platforms, this file might not contain all the keystrokes typed by the user. The typical case where keystrokes are "dropped" is when the application performs a keyboard grab, and thus does not allow the AT-SPI to get the events. XEvIE will solve this problem (and others) because it can grab events at the X server level before any toolkit or application can get its hands on them.

To check for missing keystrokes, pass the file to sanity_check.py, which performs some trivial checks for things that can obviously be broken. Note that you'll need to hand-edit the keystrokes file if sanity_check.py detects problems.

To play back a test:

• Put the desktop back in the well-known starting state as described above.
• Start Orca with logging information turned on such that Orca says what it is sending to the speech synthesizer and the braille display. Record this logging information to a different file than above.
• From a separate machine, run play_keystrokes, using the saved keystroke file.

You'll now have two separate log files from Orca that you can check for differences. Create a number of these test cases, and you can run them over night, providing more test coverage for something that typically would not get tested.

NOTE: to streamline the recording a little bit, Orca itself listens for the "Pause" key to be pressed. When a user presses "Pause," Orca will pop up a window asking for a filename for where to save the keystrokes. It will then record all keystrokes to this file until the user presses "Pause" again.

OK, so this is interesting and serves an immediate need. But, there are problems you need to know about:

• Unless XEvIE is enabled, you may not get all keystrokes.
• The record and playback is done using keycodes. While these will be the same for the same platform (e.g., Linux), they can vary between platforms. As such, keystroke files are really not portable between, say, Solaris and Linux.
• The Java platform has its own platform-independent notion of keycodes. As a result, the keycodes obtained from Java applications will typically differ that those obtained from GTK or other applications if XEvIE is not enabled. This should NOT really have an impact on testing, but it is worth noting. Note that since XEvIE will obtain keystrokes at a much lower level, this disparity should disappear.

Now, having said all this, one of my teammates on Orca, Rich Burridge, came across an announcement for dogtail, which appears to be a much more complete and sophisticated answer to what people might be looking for. Get this: "dogtail is a GUI test tool and automation framework written in Python. It uses Accessibility (a11y) technologies to communicate with desktop applications." Neat.