Declarative programming of graphical interfaces by visual examples

ABSTRACT

Graphical user interfaces (GUI) provide intuitive and easy means for users to communicate with computers. However, construction of GUI software requires complex programming that is far from being intuitive. Because of the “semantic gap” between the textual application program and its graphical interface, the programmer himself must conceptually maintain the correspondence between the textual programming and the graphical image of the resulting interface. Instead, we propose a programming environment based on the programming by visual example (PBVE) scheme, which allows the GUI designers to “program” visual interfaces for their applications by “drawing” the example visualization of application data with a direct manipulation interface. Our system, TRIP3, realizes this with (1) the bi-directional translation model between the (abstract) application data and the pictorial data of the GUI, and (2) the ability to generate mapping rules for the translation from example application data and its corresponding example visualization. The latter is made possible by the use of generalization of visual examples, where the system is able to automatically generate generalized mapping rules from a given set of examples.

REFERENCES

• 1: Alan Borning , Robert Duisberg, Constraint-based tools for building user interfaces, ACM Transactions on Graphics (TOG), v.5 n.4, p.345-374, Oct. 1986 [doi>10.1145/27623.29354]
• 2: Stuart K. Card , George G. Robertson , Jock D. Mackinlay, The information visualizer, an information workspace, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.181-186, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108874]
• 3: Cox, K. C. and Roman, G.-C.: Visualizing Concurrent Computations, in IEEE Workshop on Visual Languages, pp. 18-24 (1991).
• 4: Allen Cypher, EAGER: programming repetitive tasks by example, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.33-39, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108850]
• 5: S. K. Feiner , Clifford Beshers, Worlds within worlds: metaphors for exploring n-dimensional virtual worlds, Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology, p.76-83, October 03-05, 1990, Snowbird, Utah, United States [doi>10.1145/97924.97933]
• 6: B. N. Freeman-Benson, A module mechanism for constraints in Smalltalk, Conference proceedings on Object-oriented programming systems, languages and applications, p.389-396, October 02-06, 1989, New Orleans, Louisiana, United States
• 7: Bjorn N. Freeman-Benson , John Maloney , Alan Borning, An incremental constraint solver, Communications of the ACM, v.33 n.1, p.54-63, Jan. 1990 [doi>10.1145/76372.77531]
• 8: H. Rex Hartson , Deborah Hix, Human-computer interface development: concepts and systems for its management, ACM Computing Surveys (CSUR), v.21 n.1, p.5-92, March 1989 [doi>10.1145/62029.62031]
• 9: Tyson R. Henry , Scott E. Hudson, Interactive graph layout, Proceedings of the 4th annual ACM symposium on User interface software and technology, p.55-64, November 11-13, 1991, Hilton Head, South Carolina, United States [doi>10.1145/120782.120788]
• 10: Scott E. Hudson , Shamim P. Mohamed, Interactive specification of flexible user interface displays, ACM Transactions on Information Systems (TOIS), v.8 n.3, p.269-288, July 1990 [doi>10.1145/98188.98201]
• 11: Scott E. Hudson , Andrey K. Yeatts, Smoothly integrating rule-based techniques into a direct manipulation interface builder, Proceedings of the 4th annual ACM symposium on User interface software and technology, p.145-153, November 11-13, 1991, Hilton Head, South Carolina, United States [doi>10.1145/120782.120798]
• 12: Tomihisa Kamada, Visualizing Abstract Objects and Relations, World Scientific Publishing Co., Inc., River Edge, NJ, 1989
• 13: T. Kamada , S. Kawai, An algorithm for drawing general undirected graphs, Information Processing Letters, v.31 n.1, p.7-15, April 1989 [doi>10.1016/0020-0190(89)90102-6]
• 14: Tomihisa Kamada , Satoru Kawai, A general framework for visualizing abstract objects and relations, ACM Transactions on Graphics (TOG), v.10 n.1, p.1-39, Jan. 1991 [doi>10.1145/99902.99903]
• 15: Anthony Karrer , Walt Scacchi, Requirements for an extensible object-oriented tree/graph editor, Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology, p.84-91, October 03-05, 1990, Snowbird, Utah, United States [doi>10.1145/97924.97934]
• 16: Kurlander, D. and Feiner, S.' Inferring Constraints from Multiple Snapshots, Technical Report CUCS 008- 91, Columbia University Computer Science (1991).
• 17: D. L. Maulsby , I. H. Witten, Inducing programs in a direct-manipulation environment, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.57-62, March 1989
• 18: Brad A. Myers, Creating user interfaces by demonstration, Academic Press Professional, Inc., San Diego, CA, 1988
• 19: Brad A. Myers, Creating user interfaces using programming by example, visual programming, and constraints, ACM Transactions on Programming Languages and Systems (TOPLAS), v.12 n.2, p.143-177, April 1990 [doi>10.1145/78942.78943]
• 20: Brad A. Myers, Text formatting by demonstration, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.251-256, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108904]
• 21: Brad A. Myers , Dario A. Giuse , Roger B. Dannenberg , David S. Kosbie , Edward Pervin , Andrew Mickish , Brad Vander Zanden , Philippe Marchal, Garnet: Comprehensive Support for Graphical, Highly Interactive User Interfaces, Computer, v.23 n.11, p.71-85, November 1990 [doi>10.1109/2.60882 ]
• 22: B. A. Myers , B. V. Zanden , R. B. Dannenberg, Creating graphical interactive application objects by demonstration, Proceedings of the 2nd annual ACM SIGGRAPH symposium on User interface software and technology, p.95-104, November 13-15, 1989, Williamsburg, Virginia, United States [doi>10.1145/73660.73672]
• 23: Greg Nelson, Juno, a constraint-based graphics system, ACM SIGGRAPH Computer Graphics, v.19 n.3, p.235-243, Jul. 1985
• 24: Gurminder Singh , Mark Green, Automating the lexical and syntactic design of graphical user interfaces: the UofA* UIMS, ACM Transactions on Graphics (TOG), v.10 n.3, p.213-254, July 1991 [doi>10.1145/108541.108543]
• 25: Shin Takahashi , Satoshi Matsuoka , Akinori Yonezawa , Tomihisa Kamada, A general framework for Bi-directional translation between abstract and pictorial data, Proceedings of the 4th annual ACM symposium on User interface software and technology, p.165-174, November 11-13, 1991, Hilton Head, South Carolina, United States [doi>10.1145/120782.120800]
• 26: Tonouchi, T., Nakayama, K., Matsuoka, S., and Kawai, S.' Creating Visual Objects by Direct Manipulation, in IEEE Workshop on Visual Languages (1992 (To appear)).
• 27: J. M. Vlissides , M. A. Linton, Unidraw: a framework for building domain-specific, Proceedings of the 2nd annual ACM SIGGRAPH symposium on User interface software and technology, p.158-167, November 13-15, 1989, Williamsburg, Virginia, United States [doi>10.1145/73660.73680]
• 28: David Wolber , Gene Fisher, A demonstrational technique for developing interfaces with dynamically created objects, Proceedings of the 4th annual ACM symposium on User interface software and technology, p.221-230, November 11-13, 1991, Hilton Head, South Carolina, United States [doi>10.1145/120782.120806]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  D. Software
  D.1 PROGRAMMING TECHNIQUES

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Screen design (e.g., text, graphics, color)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

General Terms:
Design, Human Factors

Keywords:
constraints, direct manipulation, graphical user interface, layouts, programming by example, visualization