Re: 'xkill' approach to identifying GUI components on X11 ...
A comment on my previous posting asked
why not use the "xkill" method and have the tool use XQueryTree in
combination with a manual "user points at the button" approach? As you
remember, xkill lets you kill an application by clicking on the window
that a process owns. Perhaps the combination of XQueryTree-derived
information and a user pointing out the various buttons, just once,
will give you enough data on each button to determine programmatically
what can be pressed?
I had to dredge my memory banks for "xkill". The approach you say
is to do a mouse grab, change the cursor, and follow it around until
there's a click. At that point you can do XQueryTree to determine
the GUI hierarchy under the mouse.I had considered something like that ... We could obviously examine the rectangles we get from XQueryTree and even if we don't know their GUI Component class, we could manually examine the tree and learn something that way. I vaguely remember writing an application that would do screen grabs of all these rectangles that could help with the analysis.
The problem is applications written with GUI toolkits that don't show their GUI tree through XQueryTree. (e.g. Java/Swing)
In the original Xlib approach to GUI applications, each component would live within a Window. The word "Window" here doesn't mean a top-level window, but any rectangular area. They're arranged in a parent-child tree-like hierarchy, and this is the hierarchy XQueryTree shows you. The Xt based toolkits (Motif) religiously used these Window objects to contain their Widgets. If you want to read about this in more depth get the O'Reilly books on X11, volumes 2 and 4 (or else it's 1 and 3).
The easiest way to see what XQueryTree returns is with
xwininfo -tree
And it does the mouse-grab letting you select an application and dump its GUI component tree. Notice how nothing has a 'name' attached to it, just coordinates and sizes.
Unfortunately I'm not seeing today what I remembered from last year. Namely, I remember Mozilla based applications not returning much through xwininfo, but checking today (with Firefox 1.0.1, Mozilla 1.7.5 and Thunderbird 1.0, all compiled with GTK2) I see a rich tree shown in xwininfo.
The other issue with this approach is - what happens when the GUI layout changes. GUI layouts change in a couple ways - such as each revision of the application - or simply if the font characteristics change. The first means you'd have to regenerate all the mappings to the GUI component tree you're interested in. The second may be okay as the hierarchy ought to stay the same, just be rearranged a little.
(2005-03-14 11:53:05.0) Permalink Comments [0]
