SP00: VisualTAP and GUI Accessor

VisualTAP and GUI Accessor

Providing access to Graphical User Interfaces (GUIs) for blind people is the most difficult challenge facing accessibility designers today. Traditional screen readers, residing in the target system, retrieve visual information from the operating system and transform it into alternative sound, speech, or tactile representations. These strategies are becoming ineffective, however, as their capabilities are outstripped by the growing complexity of both operating systems and the ways in which visual data can be presented on a screen. Archimedes researchers have developed a new external screen access technique that breaks with tradition by performing all data retrieval and processing outside of the target system. A VisualTAP captures screen information from the target system, and a personal GUI accessor translates it into a form that is accessible to the user (see Figure 2). The Archimedes solution places emphasis on active participation by blind users. Whereas traditional screen readers analyze the visual data and present users with highly structured results, the GUI accessor presents multiple overlapping representations of the visual data and leaves much of the extraction and organization of meaningful information to the innate filtering and associative capabilities of the user's brain. Figure 3 shows this process.

The hardware and software required for recovering and processing the visual image includes:

A Frame Grabber that constructs a bitmap of the target system screen image. This requires reconstitution of the original pixel clock, and Analog to Digital conversion at speeds of up to 140-million samples per second.
Image Filters that reduce the complexity of the captured image by eliminating unwanted features, and creating separate images optimized for pattern recognition and Optical Character Recognition (OCR).
Pattern Recognition software that determines the coordinates of geometrically defined features such as window borders, scroll bars, and buttons; and nongeometric objects such as graphics, icons, and areas containing text.
A Haptic Display system that translates information produced by the Pattern Recognition software into force feedback images, enabling users to feel and trace out the shapes and locations of objects displayed on the screen of the target system.
OCR software that recovers text from areas of the screen defined by the haptic system. Tight integration of the haptic and OCR functions is necessary to achieve rapid response and allow users to synchronize what they are feeling with what they are hearing.
Cursor Tracking software that lets users determine where the cursor is located relative to other screen objects. Tracking the cursor is deceptively complex because it changes shape as it moves from region to region and even disappears from time to time. Furthermore, the hotspot may be located at different points on the various cursors.
Speech Synthesis, which provides the primary interface to blind users. When used alone, however, it can be quite tiresome. Speed and intelligibility are more important to a blind person than naturalness. Some blind people understand synthesized speech at 500 words per minute.
3D Sound Generation, which can be used to indicate properties such as color or texture of an object, and allow the user to sense the presence of objects located outside the immediate range of the haptic display.
Braille displays, which provide faster and more convenient access to text than synthesized speech in many situations. Until recently, the use of Braille displays has been limited by their high cost. Less costly designs are making it feasible to include Braille as a standard option for presenting text.
A TAP Interface that gives blind users full access to all keyboard and mouse functions on the target system through any of the standard input accessors. Additionally, combining a TAP with a VisualTAP creates a closed-loop control system capable of automatically performing many functions without user intervention. A spoken command to "click the start button," for example, would cause the system to identify the location of the start button within the current window, locate the cursor, and transmit mouse move commands to position it over the start button, then transmit a button click command.

The growing use of multimodal interfaces is creating new problems for deaf computer users. In the past, speech and sound were used to augment text printed on the screen of a computer. Now, they are being used in place of the text, particularly in low-bandwidth interfaces to the Internet such as PDAs and telephones. Archimedes researchers are investigating visual alternatives to spoken messages. A deaf accessor translates spoken or printed text into American Sign Language (ASL).

Initial experiments used VRML to create and animate 3D models of hands, faces, and torsos to produce sign language. This approach was abandoned because of disappointing results and high overheads in both the authoring and presentation phases. Animations were slow and jerky, and inaccuracies in the way secondary features such as shadows were rendered became very distracting to the users. Also, the 3D images degraded rapidly if viewed at anything but the original size and resolution.

Current research is focused on 2D animations using Macromedia Flash. A professional animation artist is creating the basic images required to create representations of a broad range of ASL (see Figure 4). The approach we have adopted eliminates unnecessary anatomical details and uses traditional animation techniques to portray movements and relative positions. Flash enables us to create real-time ASL images that can be accurately scaled for presentation on cellular phones, PDAs, computer screens, or projection screens in auditoriums.

We are frequently asked why it is necessary to bother with animated sign language instead of just printing text messages. The main reason is that English (or other printed languages) is the second language for most deaf people and it is difficult for them to read English quickly enough to keep up with normal conversation.

An interesting and potentially important side effect of the ASL accessor development is that the same techniques can be used to improve communications between any people who don't share a common language.

-- N.S.

Back to Article