An interactive system for recognizing hand drawn UML diagrams

ABSTRACT

Diagrams are widely used by software engineers to capture the structure and organization of software systems. The Unified Modeling Language (UML) is a commonly-used notation for such diagrams. We have designed and implemented a system for the on-line recognition of hand drawn UML diagrams. Input comes from an electronic whiteboard, a mouse, or a data tablet. A sophisticated segmentation algorithm groups pen strokes into symbols, coping with drawing inaccuracies that are common in hand drawn input. The system is organized around a retargetable kernel which provides a general front end for on-line recognition of any iconic notation. The kernel is extended with UML specific enhancements to segmentation, as well as UML specific glyph recognizers. A simple and intuitive graphical user interface allows the user to correct segmentation and recognition errors. Relatively messy freehand UML drawings are interpreted properly.

References

1. {1} J. Arvo. Caltech interface tools (cit). www.cs.caltech.edu/~arvo/software.html.]]
2. {2} J. Arvo. Computer Aided Serendipity: The Role of Autonomous Assistants in Problem Solving. In Proceedings f Graphics Interface '99, pages 183-192, Kingston, Ontario, Canada, 1999.]]
3. {3} J. Arvo and K. Novins. Fluid sketches: Continuous recognition and morphing of simple hand-drawn shapes. Submitted to 13th Annual ACM Symposium on User Interface Software and Technology.]]
4. {4} J. Arvo and K. Novins. Smart Text: A Synthesis of Recongnition and Morphing. In AAAI Spring Symposium on Smart Graphics, pages 140-147, Stanford, California, USA, 2000.]]
5. {5} L. Chen and P. Yin. A System for On-line Recognition of Handwritten Mathematical Expressions. Computer Processing of Chinese and Oriental Languages, pages 19-39, June 1992.]]
6. {6} Michael Thomsen Christian Heide Damm, Klaus Marius Hansen and Michael Tyrsted. The knight project. www.daimi.au.dk/~knight/.]]
7. {7} W. Citrin and M. D. Gross. Distributed Architectures for Pen-Based Input and Diagram Recognition. In ACM Conference on Advanced Visual Interfaces '96, 1996.]]
8. {8} E Do and M. D. Gross. Thinking with diagrams in architectural design. www.mrccbu.cam.ac.uk/projects/twd/discussion-papers/architecture.html.]]
9. {9} J. Rumbaugh G. Booch and I. Jacobson. The Unified Modeling Language User Guide. Addison-Wesley, 1998.]]
10. {10} D. Elliman G. Hutton, M. Cripps and C. Higgins. A Strategy for On-line Interpretation of Sketched Engineering Drawings. In Fourth International Conference on Document Analysis and Recognition, pages 771-775, Ulm, Germany, 1997.]]
11. {11} G. Goldman and S Zdepski. Grids in Design and CAD. In Proceedings ACADIA 91 - Reality and Virtual Reality, pages 33-43, Los Angles, California, USA, 1991.]]
12. {12} G. Goldman and S Zdepski. Grids in Design and CAD. In Proceedings ACADIA '91 - Reality and Virtual Reality, pages 33-43, Los Angles, California, USA, 1991.]]
13. {13} M. D. Gross. Why can't cad be more like lego? Automation in Construction Journal , 1996.]]
14. {14} J. A. Jorge and M. J. Fonseca. A Simple Approach to Recognise Geometric Shapes Interactively. In 3rd IAPR International Workshop on Graphics Recognition, GREC'99, Jaipur, India, 1999.]]
15. {15} H. Kojima and T. Toida. On-line Hand-drawn Line-figure Recognition and its Application. In 9th Intl. Conf. on Pattern Recognition, pages 1138-1142, Rome, Italy, 1988.]]
16. {16} A. Kosmala and G. Rigoll. Recognition of On-Line Handwritten Formulas. In 6th International Workshop on Frontiers in Handwriting Recognition, pages 219-228, Taejon, Korea, 1998.]]
17. {17} Edward Lank. Describing diagram recognition systems. available at www.cs.queensu.ca/~lank/dr.process.ps.gz.]]
18. {18} R. Pooley and P. Stevens. Using UML - Software Engineering with Objects and Components. Addison-Wesley, 1998.]]
19. {19} K. Novins S. Smithies and J. Arvo. A Handwriting-Based Equation Editor. In Proceedings f Graphics Interface '99, pages 84-91, Kingston, Ontario, Canada, 1999.]]
20. {20} M. V. Stückelberg and D. Doermann. On Musical Score Recognition using Probabilistic Reasoning. In 5th International Conference on Document Analysis and Recognition, pages 115-118, Bangalore, India, 1999.]]
21. {21} E. Valveny and E. Martí. Application of Deformable Template Matching to Symbol Recognition in Hand-written Architectural Drawings. In 5th International Conference on Document Analysis and Recognition , pages 483-486, Bangalore, India, 1999.]]
22. {22} J. Coronado Y. Dimitriadis and C. de la Maza. A New Interactive Mathematical Editor, Using On-line Handwritten Symbol Recognition, and Error Detection-Correction with an Attribute Grammar. In First International Conference on Document Analysis and Recognition, pages 242-250, Saint Malo, France, 1991.]]
23. {23} R. Zannibbi. Recognition of mathematics notation via computer using baseline structure. Master's thesis, Queen's University, 2000.]]

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Interest point and salient region detections

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Graphical user interfaces

• Software and its engineering

  • Software creation and management

    • Software development techniques

      • Flowcharts

  • Software notations and tools

    • General programming languages

      • Language types