d
Figure 1: Sample UML diagram
Technical diagrams are an inescapable part of professional life. In the IT (information technology) field, advancement often involves the ability to analyse and design systems, requiring the preparation and interpretation of diagrams. One standard vehicle for achieving this in the object-oriented community is with the Unified Modeling Language (UML). The UML is a visual communication tool which conveys non-visual (structural) information about program architecture. Where students have vision impairments, particularly blindness, alternative communication mechanisms need to be discovered that will enable such students to understand the concepts and develop into fully capable IT professionals. This paper contains a review of techniques and products that may enable blind programmers to 'read' UML diagrams.
The case of online teaching explored in this paper arose out of experiences of teaching an intermediate-level Java subject that involved a limited subset of the UML, namely class diagrams, object diagrams, sequence diagrams and dynamic modeling (state transition diagrams). One of the students was completely blind. He came to classes led by his guide dog. He used a laptop and earphones so that a screen reading program could read to him the content of lecture slides, documents and other electronic teaching materials. Much of the communication process involved the use of Blackboard, an online course management program.
This paper follows an 'action research' paradigm. It is a reflective response to the teaching and learning situation experienced. Whilst describing briefly the experiences, the focus is on helping both educators and students faced with similar circumstances in the future to benefit from the lessons learnt in this situation. This paper describes the possibilities discovered for communicating UML and suggests ways for others who need to communicate technical diagrams non-visually. The original intent of the search was to catalogue all devices and techniques that could be used, however it became clear early in the process that despite poor availability, there are many techniques and technologies; too many to cover in a single publication. The scope was then narrowed to include only one specific situation: communicating UML diagrams to a Braille-literate totally blind student in an educational context.
UML diagrams communicate the structure, function and interrelations between parts of a computer program. UML allows the programmer to make sense of chaos; to recognise patterns in complex systems; and to understand the relationships between objects.
According to the Object Modeling Group (OMG 1997), UML must be visual. This is expressed in the first goal of UML as described in an early OMG press release:
"Provide users with a ready-to-use, expressive visual modeling language so they can develop and exchange meaningful models."
"The UML is a language for visualizing, specifying, constructing, and documenting the artifacts of a software-intensive system..."
A diagramming technique for the blind should enable blind programmers to visualize, specify, construct and document a software system. If we interpret these terms in the context of a sighted person, we could say:
This example may help readers understand some of the complexities involved in the communication of a UML diagram. The example SWOON UML diagram [1] shown in Figure 1 is taken from lecture material. Distributed via the online course management system, a vision impaired student would need to render this diagram in one of many possible technologies, in order to be able to correctly interpret it’s meaning.
Readers of the paper not familiar with UML should note that the aim is not to focus on the particulars of UML, but rather on the complexities of the communication process, where diagrams are involved. There are tools that will convert the diagram in Figure 1 to various alternate formats that can be read by the vision impaired. However, there is always loss of information in the translation process. By way of analogy, readers familiar with another language watching a television program with English subtitles, will know that what was said and what the subtitle communicates are not always the same; the translation while accurate has lost some of the meaning inherent in what was spoken. Similar information loss occurs when translating a diagram. This diagram is chosen to show on the one hand that diagram rendering technologies can render diagrams reasonably well. However, it also serves to illustrate the short-comings of some technologies over others.
Figure 1 depicts a simple applet. In lectures, students learn how to develop GUIs (graphical user interfaces); both applets and graphical applications. In this example, a design is depicted for a bank account applet. Aside from rendering the diagram, students are expected to learn some key things about design through this diagram. They see in it examples of inheritance, of implementing interfaces, and of following good architecture; in this case that of the Model, View, Controller architecture is shown, where presentation code is separated from the underlying data (the Account class in Figure 1). The diagram is used early in the teaching of GUI principles and therefore shows centralised control, with only the applet itself ‘listening’ for events to do with it’s components. Later in the subject, decentralised examples tease out further design and implementation issues.
All these aspects are higher-level-of-abstraction issues, that go beyond a merely accurate representation of the elements of the diagram. Of concern in the pedagogical process is that students with vision impairments should not only receive an accurate translation of the diagram, but also that there be ways of translating or communicating these and other higher level abstractions. In one sense, Figure 1 is itself an abstraction, in that instance variables and methods are not shown. Already this diagram would convert to A3 size using several of the technologies available to people with vision impairments. Were instance variables and methods added, this diagram would convert into a diagram too large to be rendered practically.
An additional point to make about Figure 1 is that as is typical of many diagrams and much communication, there are ambiguities and inaccuracies. That is: while not deliberately attempting to mislead students, part of the pedagogy may be to get students thinking about the design and improvements that can be made. Similarly, teachers (and designers generally) can get things wrong the first time and as a result, diagrams can go through several modifications before they are in a final, possibly complete, state.
In the case of Figure 1, the "2..3" is ambiguous. It could relate to one of 2 different arrows. Either the BankAccountApplet is associated with at least 2 and possibly 3 JTextFields and the JPanel is associated with only one JLabel, or it is the other way around? The inaccuracies relate to some of the numbers on the diagram; they are deliberately inaccurate to serve as a means of teaching students how the diagram relates to a GUI. Students see what the finished GUI looks like (not shown in this paper) and soon begin to question the diagram in Figure 1. That process requires students to compare two different types of diagrams and understand how they relate to each other.
The information on diagramming techniques discussed in this paper was obtained by following up suggestions made in response to questions posted on newsgroups such as WebAim, JavaAccess, BlindProgramming, and in person at an open day of the Vision Australia Foundation. Supplier’s web sites, catalogues and in some cases, shops were investigated.
Responses to initial research included references to various products and approaches, and requests for answers; i.e. 'When you find out, let me know'. Another response that several people sent was that they were limited in their professional career advancement because of poor communicating skills where diagrams were concerned. For instance, a blind programmer said she was stuck in terms of advancement in her job, because of inability to communicate visual designs, when that is a standard vehicle of advancement from programming to analysis.
Suggestions of different techniques led to this review of technologies and methods. Many of these approaches were tried out in the context of teaching UML to a blind student, and comments on their effectiveness are included where appropriate[2].
Non-visual diagramming techniques can be easily divided into tactile and non-tactile categories. The most prolific technologies concentrate on converting a visual diagram (usually stored in a computer) into a tactile diagram on paper or on a Braille display. At the other end of the spectrum, a trained interpreter can describe a UML diagram verbally or in text. That spectrum is here described in five categories. In the appendix to this paper various constraints to the adoption of a diagramming technology/technique are discussed. Once the user has a clear idea of which constraints are paramount, the selection of a diagramming method can be made using Table 1.
A sighted person can produce a one-off tactile image from scratch using manual tools. Due to the inherent problems of clutter in a tactile diagram, the use of craft or freehand raised drawing techniques can be beneficial in building up a complex understanding of a technical diagram. Initially, the key elements of the diagram without complex labeling or information can be constructed and presented to the blind user. This base diagram can then be manipulated and expanded to include finer detail and larger amounts of information. Cost less than $500.
These are used in conjunction with appropriate graphic authoring/translation software. Braille embossers can be used to create diagrams made up of embossed dots. Appropriate software can convert a screen image or stored image file into a dot pattern by substituting appropriate Braille characters for parts of the diagram. For example, the Braille characters for ‘c’ and ‘l’ could be repeated to represent horizontal and vertical lines respectively.
Some embossers have a graphics mode which reduces the spacing between the embossing pins, allowing higher resolution of graphic images. Costs range from US$3,000 to US$10,000 depending on the resolution of the embosser.
With stereocopying technology, lines, dots and symbols are printed on to microcapsule paper (‘swell’ paper) and ‘developed’ using a stereocopying machine (‘tactile image enhancer’ or ‘fuser’). A stereocopier is a device which heats the paper wherever printing has occurred. As the paper under the ink heats up, small ‘microcapsules’ inside the paper swell up and produce a tactile image of the printed shape. Tactile image enhancers cost between US$100 and US$1,300. The paper is expensive at about US$1.00 per sheet. A commercial service in Melbourne charges AUD$18 per A4 page for stereocopying diagrams. Stereocopiers can be used by blind people. Diagrams can be altered using a heat pen (similar to a small soldering iron).
Various do-it-yourself methods also exist for getting a tactile image from printed paper (RNIB).
A sighted person used a Tiger embosser to produce tactile graphics for out blind student. It was useful to use the various dot heights to mark and identify multiple areas within flow charts and diagrams with many regions. Stereocopiers are useful when converting UML diagrams to tactile form from textbooks or lecture notes. They can be difficult for a vision impaired person to modify print diagrams effectively before tactile production. Microcapsule paper diagrams can be difficult to interpret if the paper is exposed to extensive heat during printing or photocopying. The background of the diagram raises slightly and can make some tactile features difficult to distinguish.
If Braille is to be included with the diagram, tactile image authoring software can be used to format diagrams for embossing or printing onto microcapsule paper.
A specialised output device is connected to a computer which allows the user to examine graphics on the screen. The device converts a part of the screen image to tactile form by use of software and retractable pins mounted in an array on the output device. Costs for these types of technologies vary considerably between US$700-8,000 depending on the size of the display.
This approach is particularly useful for complicated diagrams or diagrams where Braille annotations are not practical (either the user or the 'author' does not know Braille).
A tactile image overlay is placed on a pressure-sensitive graphics tablet. The graphics tablet is connected to a computer running software that generates audio descriptions and other audio content. The user examines the tactile image and can select particular parts of the image which trigger the audio content allocated to a particular place on the image. These devices are expensive and complicated to set up.
A low-cost alternative involves combining a verbal description (often played back on an ordinary tape recorder) with a tactile diagram made by stereocopying or another 'low cost' method. The tactile version of the diagram is studied in conjunction with the audio description.
Tactile/Audio diagrams were proven very effective when exploring UML class diagrams. Collaborations and relationships were described verbally and programmed for each region of the diagram.
Combining a taped audio description with a tactile diagram was also very useful. Identifiable regions and markers could be described within a diagram to reinforce the concepts conveyed by the diagram. This was especially useful when producing tactile graphs of mathematical functions.
The meaning of the UML diagram can be described as an alternative to a visual representation. This approach is suitable for meetings, conversations and communication by e-mail.
This approach works well for diagrams composed of nodes and leaves. Each node is described as a room which is described to the user. The room has doors (leaves) which lead to other rooms. Each door (leaf) has it’s own characteristics that can be described. This approach was used for text-based adventure games in the 1970’s and 1980’s.
Experienced computer programmers can use computer-like instructions to describe objects, shapes and lines to build up a verbal description of a diagram. This is excellent for communicating concepts usually conveyed by flow charts. As this is a text based medium, it is very easy to convey the flow of a program quickly and effectively. This can be less effective if large diagrams are to be interpreted, as relationships between components can be lost. Generally sighted people find this medium time consuming to interpret.
The actions and variables of different objects can be described in text form using CRC cards (Rubin, 1998). This is very useful for working at the component level of a system, for example, objects and classes. It can be difficult to get an overall concept of the system especially when the system is large and complex. CRC cards can be very effective if used in conjunction with a tactile diagram showing a high level view of the system.
A sighted user or designer can describe a diagram to a blind person, however this is a lot harder than most people expect it to be. The interpreter has to understand object oriented modeling and be able to communicate it verbally.
UML diagrams represent the structure and interrelations within a software program.
UML diagrams do not represent any physical layout or spatial relationships. It therefore follows that UML diagrams do not represent visual information. Rather, UML allows the presentation of complex data with a fine level of detail in a way that is convenient to sighted programmers. The challenge for developers of assistive technology[3] is to make UML representations of software accessible to vision impaired or blind programmers. This is not the same thing as developing ways of converting printed or on-screen diagrams into tactile diagrams.
The appendix contains a check list of selection criteria or constraints which, together with Table 1 (for teaching UML in this scenario) or Table 2 (for non-visual diagramming scenarios not discussed in this paper) can be used to aid the selection of an appropriate technology. In the following discussion, different technologies are evaluated in terms of their ability to communicate UML diagrams over the world wide web to students.
A sighted user can make up a tactile diagram based on the contents of a printed UML diagram. Because the process is manual, a lot of information can be conveyed on one diagram. Shaded areas can be represented as textured areas. Annotations and captions can be represented as Braille text. The process is slow, entirely manual and the effectiveness of the result may not be immediately understood by a blind user. Building up a diagram while the blind user is present and gradually increasing detail dramatically reduces the initial cognitive load on the blind user, enabling an improved understanding of complex representations.
UML diagrams such as Figure 1, could be converted this way. However, it is not recommended for diagrams that change frequently. Diagrams produced by manual methods cannot be transmitted electronically and are therefore not suited to online delivery.
Technologies for making tactile diagrams with minimal human intervention are a popular area of development. Braille embossers (through a layer of software and mechanical alterations to the embosser) can be coaxed into producing diagrams composed of Braille dots; the user has to ‘join the dots’ in order to reconstruct the diagram mentally. A layer of software can also be used to convert alphanumeric text to Braille text, placing it on the same places in the diagram.
This approach suffers from the resolution limitations of both the Braille embossers and the fingertips of the blind users. To be readable, Braille text has to be much larger than alphanumeric text. Embossed diagrams have to be much larger than those produced using normal ink on paper because the tactile resolution of the lines or dots is much lower than that of the eye. Braille embossed diagrams are very useful when producing diagrams with very few labels or textual information.
However, the inherent problem with this technology is the limitations placed on the design and layout of the diagram being produced. Embossing technology is only capable of low resolution production, often with straight line segments being used to build the diagram (National Centre for Tactile Diagrams: Making tactile graphics). It is very difficult to manipulate technical diagrams in such a way as to ensure the diagram fits on the page, while taking into consideration the clutter within the diagram. Often, the diagram can be segmented and rendered on multiple sheets, however, this method can further hinder the process of understanding relationships between regions of a technical diagram. If the diagram is annotated, the resolution problem increases. Space has to be allowed for alphanumeric text to be replaced by much bigger Braille text.
High-resolution graphics-mode Braille embossers like the Tiger improve the rendering of lines and symbols (with dots that overlap, producing something close to a commercially embossed diagram), but fail to address the problem of annotation. If the text is converted to an equivalent size Braille font, it will be too fine to be read by most Braille-literate users. If the Braille text is embossed using a sufficiently large font, it will overlay other Braille annotations and confuse the reader, who will no-longer be able to identify which annotations relate to specific parts of the diagram. To reduce this clutter, labeling can be reduced to a numbering system and a key can be provided as an attachment to ensure that a greater proportion of the diagram can be rendered on one page.
Stereocopiers have the potential to improve the resolution of tactile graphics dramatically because they use conventionally printed diagrams as the source of the tactile graphic. The most important rule to remember when using this technology is to never expect to produce a tactile diagram from an unmanipulated print or electronic copy of the diagram (National Centre for Tactile Diagrams: Making tactile graphics). Printed or electronic diagrams are rendered using much higher resolutions and finer detail than can be produced tactually. Labeling is often small and does not translate well to a tactual context. It is important to remove as much extraneous information as possible and to enlarge the diagram to ensure ease of navigation and identification of regions within the diagram. Braille font labeling can be added to electronic diagrams before printing and can aid the blind user in identifying key areas of the diagram.
After production it is possible to expand the diagram using a heat pen which can be useful when refining and adding detail to the diagram (National Centre for Tactile Diagrams).
UML diagrams, as in Figure 1, contain a lot of text in boxes, so there will be a problem. There is also the problem of frequently changing diagrams. New printouts (via embosser or stereocopier) are required each time significant changes are made, increasing the costs of production.
UML diagrams contain a high level of complexity and a lot of detail. A sighted programmer can stand back from a diagram to get the ‘big picture’ - the patterns of relationships between objects / classes. The same user can look closer at a particular object on the diagram and read a list of instance variables, private methods, public methods and so on. Often these details are coded with extra symbols or changes in font to speed up interpretation. The sighted user is usually unaware of the process of zooming in to get the detail.
For the blind user, all ‘vision’ is set permanently on zoom. The blind user has to try to interpret a diagram while ‘looking’ through an aperture the size of a fingertip, with a resolution measured in millimeters. To ‘see’ the entire diagram, a blind user has to scan the diagram while building a mental image one fingertip at a time. This is an enormous burden on cognitive ability and memory[4]. The ‘big picture’ interpretation of an embossed diagram is very difficult to obtain for a blind user.
One possible solution is to make a set of tactile diagrams, ranging in size, detail, and area of focus. This is prohibitively expensive (especially if the diagram has a lot of text in little boxes). Tactile displays are a possible solution. These displays act as a screen or VDU (visual display unit) for the blind user. The ‘screen’ is a small array of pins which rise and fall indicating the information currently in focus on the ‘big screen’. The advantage for the blind user is that the area of focus can be easily moved, zoomed or ‘unzoomed’. The user’s finger never leaves the tactile display, but the information represented there may contain fine detail (when zoomed) or the overall appearance of the entire diagram. The resolution limits of fingertip reading are still present, but the user can choose the level of zoom, and can examine complex parts of a diagram much more easily.
In Figure 1 this approach would be helpful in moving from the high level of abstraction shown, to a lower one that includes instance variables and methods.
Tactile displays are expensive and are beyond the reach of most computing students. When applying this technology to the teaching of UML, it is also worth considering that UML diagrams have a lot of text. If the text is stored as an image, it will not be converted to Braille by any software layer present. The user has to be able to read tactile alphanumeric characters, which are ambiguous at best. Space / resolution considerations still apply to alphanumeric text converted to Braille.
It is important to consider the effects of improved audio technology, scanners, optical character recognition and computing power have made reading possible without using Braille. It has been estimated that only about 12% of blind people know Braille. If the user is not a Braille reader, the most effective way to communicate textual information can be via an audio description. In it’s simplest form, audio description can be recorded onto a tape which can be played by the user while interpreting the tactile diagram. It is very important to describe prominent features of the diagram and the relationships between regions. A more effective approach is to provide the user with an interactive audio/haptic (touch) experience. This can be achieved by the use of a touch tablet connected to a PC with software allowing regions of a diagram to be programmed with descriptive audio material. The tactile diagram is placed upon the touch tablet giving the user the ability to touch specified regions within the diagram and receive an audio description of that region and it’s relationships to surrounding regions.
Audio/Haptic systems are expensive and have not been a widespread commercial success. Most systems have been produced in small numbers. Despite these limitations, the audio/haptic approach provides a potentially effective way of communicating large amounts of complex information.
This technology provides the best of both worlds. The usefulness of a tactile diagram is dramatically enhanced by audio description. A technical diagram can be produced with very little information contained on the diagram itself. Clutter is not a problem. The combination of the audio and tactual input can greatly reduce the cognitive load on the blind user and can improve understanding of technical representations.
With regards to the context of this paper, UML diagrams can be successfully conveyed this way. The stumbling block is the cost and time needed for setting up the diagram. Diagrams can not be converted from ‘sighted’ to tactile/audio automatically. On the other hand, audio descriptions do cater more easily to diagrams that frequently change. The inconsistencies and ambiguities in Figure 1 can most easily be dealt with using this type of technology.
If audio/haptic presentations are so good, and tactile diagrams are so bad, why not use audio by itself? Text descriptions are easily converted to alphanumeric text, Braille, or voice. Written text can be scanned, converted to Braille, alphanumeric text or voice. The technologies are mainstream, compared to tactile diagrams. Information can be transmitted quickly and cheaply using e-mail and the world wide web. A variety of approaches can be used to describe the relationships between, and the contents of UML objects.
Although this approach works well for blind programmers, it does not fit well into the practices, standards and abilities of sighted programmers. UML diagrams are supposed to be diagrams - not audio presentations or verbal tours through a software design. UML users expect to get information on many levels simultaneously - relationships, structure, details; even standards are expressed immediately to a sighted programmer when they see a UML diagram. Sighted programmers are not capable of the long term memory and cognitive pattern building abilities that blind programmers are forced to have. A blind programmer using this technology can not expect advancement is a software company who insist on standard UML notation.
Whether teaching online or working professionally, there are times where complex, technical diagrams need to be communicated effectively between people. The above discussion shows the complexities involved, even when that communication process is limited to one between a visually-able teacher and a blind student.
In the context of this investigation, the best way of communicating UML to a blind student is to use an audio/haptic approach. This approach can be used cheaply (using audio or text played through a web browser and tactile images created using stereocopying) and has the benefits of providing detailed information and 'big picture' views of software structure, without the need for Braille literacy. The main difficulty is the time and expertise needed to convert 'visual' diagrams to a form which can be meaningfully described both audibly and tactually. Conversion cannot be done automatically. Finding the resources and technical support to do the conversion will present a problem for many academics.
The problem is not with the blind student/programmer. UML is not essential to the design and creation of good object-oriented software. It is a means to an end, and not an end in itself. If the software industry is really serious about issues of accessibility, equity and non-discrimination, it needs to embrace alternatives to standard visual UML. Other parts of the IT world have already made steps toward accommodating vision impaired users. The world wide web consortium have produced guidelines for improved accessibility, and implemented standards which encompass non-visual ‘display’ of web pages. Microsoft have adopted a set of object models which allow accessibility software to ‘read’ any displayable object, and Java applications and applets provide extra methods specifically intended for use by non-visual display technologies. UML needs to catch up.
Refer to Table 1 for technologies based on the factors above.
| Method | Type of information | Type of user | Skills/Senses needed | Other constraints | On-line delivery | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Blind author | Sighted reader | |||||||||||
| Edit-able | Relat-ional | Com-plex | Linear | Braille | Audio | Cost | Quan-tity | Dur-able | ||||
| Manual tools | Yes | Yes | No | Yes | Yes | Yes | No | No | Low | Small | No | No |
| Embosser | No | Yes | No | Yes | Yes | No | Yes | No | Medium | Med-ium | No | Yes |
| Stereo-copy | Yes | Yes | No | Yes | Yes | Yes | Yes | No | Medium | Small | No | Yes |
| Tactile Display | Yes | Yes | No | Yes | No | No | Yes | No | Medium | One | No | Yes |
| Audio-Haptic | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Low-medium | One-many | Yes | Yes |
| Audio-Text | Yes | No | No | Yes | Yes | Yes | No | Yes | Low | High | No | Yes |
| Method | Type of information | Type of user | Skills/Senses needed | Other constraints | On-line delivery | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Blind author | Sighted reader | |||||||||||
| Edit-able | Relat-ional | Com-plex | Linear | Braille | Audio | Cost | Quan-tity | Dur-able | ||||
| Voice (Seeing-With-Sound) | No | No | No | No | Yes | No | No | Yes | Low | No | No | Yes |
| allCLEAR (flow-charting software) | Yes | Yes | No | Yes | Yes | Yes | No | No | Low | No | No | Yes |
| UML program (GraphViz, UMLGraph) | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | None | Med-ium | No | Yes |
| Graphics Software (Tactile Graphic Designer) | Yes | Yes | No | Yes | No | No | Yes | No | Medium | Yes | No | Yes |
| Vacuum forming (RNIB) | No | Yes | No | Yes | No | Yes | Yes | No | Medium | Yes | No | No |
| Photo etch (RNIB) | No | Yes | No | Yes | No | Yes | Yes | No | Medium | No | Yes | No |
| Screen print / letterpress (RNIB) | No | Yes | No | Yes | No | Yes | Yes | No | Medium | High | No | No |
| CAD/CAM, Rapid Prototyping (RNIB) | Yes | Yes | Yes | Yes | No | Yes | Yes | No | High | High | Yes | Yes |
[1] SWOON is a subset of UML developed by Swinburne for teaching purposes.
[2] All anecdotal examples given in this paper reflect experiences with a particular technology by one of the authors, who is totally blind. There is no suggestion that these experiences are typical of other, let alone all, blind users.
[3] Technology designed to allow users with disabilities to function in an environment geared towards able-bodied users.
[4] Reading text as Braille does not require the same extreme of mental acuity because writing/speech is linear. Humans have been practicing understanding sentences uttered one word at a time since birth, and are well adapted to it.