Reverse Engineering Accessibility

I thought I'd give you a glimpse of the kind of problems I'm working on daily, and how I have to go about solving them. If you like solving puzzles, this can be very interesting. I'll leave you to decide whether there should be a better way to do this.

As you probably know, I'm in the Accessibility Program Office (APO) and our charter is to help make all of the products that we ship, accessible to people with disabilities.

This quite often involves working with other product groups in different parts of the company and also with external open source developers.

One piece of software that we have started within the APO is Orca. It's a screen reader/magnifier to help make our desktop accessible to blind and low vision people. You can read more about Orca on Will Walker's blog.

Our goal here is to make all the JDS/GNOME desktop accessible. This will also include making sure applications like Firefox and the StarOffice/OpenOffice suite work too.

As you might know, accessibility support is fully integrated with the Gtk+ toolkit that GNOME applications are built with. If accessibility has been turned on, then as you run your GNOME/Gtk+ applications, there are literally thousands of events flying around that are describing what's going on. A screen reader such as Orca is listening in, and trying to make sense of all this and present it to the user in a useful way.

For example, you get events for when a new window appears on the screen, when focus changes from one component to another, when the cursor moves in a text area, and much much more.

Blind users don't use the mouse. They rely on the keyboard. Everything that you can do via the mouse also needs to be doable via just the keyboard, and in a straight forward, intuitive and consistently repeatable manner. You'd be amazed how often this isn't the case.

Currently I'm looking at getting Evolution, the mail, calendar and contacts application, working with Orca. Mail support seems to be working fairly well now (Orca output is in speech and braille), and I've started looking at calendar support.

Anybody who's looked at Evolution calendar (and here's a screenshot of an earlier version), will recognize the day view in the middle, divided into half hour increments.

Gtk+ comes with lots of standard widgets, and the accessibility support for these is fairly good. Evolution has decided to create its own custom components for some things such as the day view.

When the user Tab's into that pane, you want to let them know what appointments are available and provide details on each appointment; say a start and end time and a summary of what the appointment is about.

For example, if I setup my calendar with two appointments, the first one starting at 10am for one hour with a summary of "Weekly Staff Meeting" and the second one starting at 1pm for thirty minutes with a summary of "Harry 1:1", using the default information provided by Evolution, Orca is able to speak:

'Day View: Thursday 16 Dec 2005. It has 2 events. calendar view'
'Calendar Event: Summary is Weekly Staff Meeting.    calendar event'
'grayed'

If I now Tab to the second appointment, I get the following spoken:

'Calendar Event: Summary is Harry 1:1.    calendar event'
'grayed'

The trouble with this is that the start and end time for those appointments is simply not available, so it can't be spoken. Those time stamps to the left of the appointments are drawn on a canvas with Pango so it's not possible to get to them. The user could type Control-O whilst the focus is on one of those appointments. This would open up the Calendar Appointment dialog. Unfortunately, it appears you've can't navigate with the keyboard, to the fields in this popup that would give you the start and end date and time.

Now as we find these problems, we file bugs against Evolution and hopefully they will be fixed in a future release. We could also file an enhancement request asking (nay, pleading) that the day view would be a lot more accessible if it was a table with column headers and rows of appointments, but somehow I don't expect the Evolution team to decide to change this, as the current component would look nicer.

So what can you do?

Well, in order to fix this, you have to reverse engineer the Evolution application component hierarchy from the at-spi events you've received.

Orca is written in Python. The clever design allows you to write a custom script to handle a specific application. I'm working on one called Evolution.py for Evolution. This script is subclassed from a default script that is used to handle applications that don't have (or need) their own scripts.

Within your custom script, you can override whatever methods you want. For example, I can write my own onFocus() method, that gets called whenever a component in Evolution gets focus.

When that happens, you will get a Focus event. From that, you are able to get a handle to the component that received the focus, and from that, you can walk your way back up the component tree until you get to the root (which has a role of "application"). Knowing the role of the current component, plus the role of its parent, and its parent, you can (usually) determine exactly which component in the application got focus.

For example, if you've tabbed into the calendar day view pane onto an appointment, or are tabbing between appointments within that pane, then the hierarchy looks like:

Calendar Event        - an Evolution custom component
Calendar View         - an Evolution custom component
filler 
panel

It gets better. If you Tab into the calendar day view pane on a day that doesn't have appointments, then the hierarchy looks like:

table
Calendar View
filler
panel

If you use the arrow keys to navigate around within the calendar day view, then the hierarchy looks like:

unknown
table
Calendar View
filler
panel

After a bit of investigation, I was able to determine that the Calendar View contains a table with N children in it, one for each time increment period in the day (increments can vary for 5 minutes to 1 hour). If a day has appointments, then that Calendar View will also have one Calendar Event child for each appointment.

You can determine the extents (x/y position, width and height) for each component). By looking at the day view pane, you can see that the appointments are placed at the same y coordinate at one of the children in the table. From this, you can determine what the start time of that appointment was.

Using the height of one of those N children, you can determine how "high" the time increment is. From this, you can then, for each appointment, work out the end time. Given all this information, we could now use it to provide better speech output for those two appointments:

'Calendar appointment. Starts at 10am. Finishs at 11am. Summary is Weekly Staff Meeting.'

and

'Calendar appointment. Starts at 1pm. Finishs at 1 thirty pm. Summary is Harry 1:1.'

This is just one such scenerio. Most of the code in the scripts is not this convoluted as the applications tend to use standard Gtk+ components.

Are we having fun yet?

[Technorati Tag: Accessibility]

[Technorati Tag: Reverse Engineering]

( Dec 16 2005, 09:09:12 AM PST ) [Listen] Permalink