Natural Contextual Reasoning for End Users

Abstract

The realization and deployment of the Internet of Things require providing to non-programmers some level of programmatic control for tailoring system behaviour to their context and needs. We introduce a simple context-range semantics (CRS) and a context-range editor (CoRE) that support end users formulate and understand logical expressions regarding context. The editor builds on two key ideas (a) contextual information is used to evaluate and minimize logical expressions; (b) logical expressions are presented in a disjunctive normal form (DNF) thus applying a principle established in mental model theory. User tests reveal situations in which the theory regarding the intuitiveness of the DNF needs to be extended with a new element: Logical terms are easier to comprehend and formulate when grouped according to their semantic affinity. We report two experiments that demonstrate the intuitiveness of this approach and how it improves performance of non-programmers in specifying context sensitive system behaviour.

References

1. G. Abowd, Russell Beale, A. Dix, and J. Finlay. 1996. Human-Computer Interaction. Graduate computer science course offered at Georgia tech, winter.
2. Gregory D. Abowd and Elizabeth D. Mynatt. 2000. Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput.-Hum. Interact. TOCHI 7, 1 (2000), 29--58.
3. Saeed Aghaee and Cesare Pautasso. 2014. End-user development of mashups with naturalmash. J. Vis. Lang. Comput. 25, 4 (2014), 414--432.
4. Carmelo Ardito et al. 2014. User-driven visual composition of service-based interactive spaces. J. Vis. Lang. Comput. 25, 4 (August 2014), 278--296. DOI:https://doi.org/10.1016/j.jvlc.2014.01.003
5. Louise Barkhuus and Anind Dey. 2003. Is context-aware computing taking control away from the user? Three levels of interactivity examined. In Proceedings of the IEEE Ubiquitous Computing (UbiComp’03). Springer, 149--156.
6. Victoria Bellotti and Keith Edwards. 2001. Intelligibility and accountability: Human considerations in context-aware systems. Hum.--Comput. Interact. 16, 2--4 (2001), 193--212.
7. Alan F. Blackwell. 2002. First steps in programming: A rationale for attention investment models. In Proceedings of the IEEE 2002 Symposia on Human Centric Computing Languages and Environments. IEEE, 2--10.
8. Alan F. Blackwell and Rob Hague. 2001. AutoHAN: An architecture for programming the home. In Proceedings of the IEEE Symposia on Human-Centric Computing Languages and Environments. IEEE, 150--157.
9. Ruth M. J. Byrne and P. N. Johnson-Laird. 1992. The spontaneous use of propositional connectives. Q. J. Exp. Psychol. 45, 1 (1992), 89--110.
10. Keith Cheverst, Hee Eon Byun, Dan Fitton, Corina Sas, Chris Kray, and Nicolas Villar. 2005. Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Model. User-Adapt. Interact. 15, 3--4 (2005), 235--273.
11. Eric Christensen, Francisco Curbera, Greg Meredith, and Sanjiva Weerawarana. 2009. Web services description language (WSDL) 1.0. (September 2009). Retrieved January 15, 2017 from http://xml.coverpages.org/wsdl20000929.html.
12. Philipp Cimiano, Andreas Hotho, and Steffen Staab. 2005. Learning concept hierarchies from text corpora using formal concept analysis. J. Artif. Intell. Res. 24 (2005), 305--339.
13. Jose Danado and Fabio Paternò. 2014. Puzzle: A mobile application development environment using a jigsaw metaphor. J. Vis. Lang. Comput. 25, 4 (2014), 297--315.
14. Anind K. Dey, Raffay Hamid, Chris Beckmann, Ian Li, and Daniel Hsu. 2004. A CAPpella: Programming by demonstration of context-aware applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’04). ACM, New York, NY, 33--40. DOI:https://doi.org/10.1145/985692.985697
15. Anind K. Dey, Timothy Sohn, Sara Streng, and Justin Kodama. 2006. iCAP: Interactive prototyping of context-aware applications. In Proceedings of the Pervasive Computing, Kenneth P. Fishkin, Bernt Schiele, Paddy Nixon, and Aaron Quigley (Eds.), Lecture Notes in Computer Science. Vol. 3968. Springer, Berlin, 254--271.
16. W. Keith Edwards and Rebecca E. Grinter. 2001. At home with ubiquitous computing: Seven challenges. In Proceedings of the Ubiquitous Computing (Ubicomp’01), Gregory D. Abowd, Barry Brumitt, and Steven Shafer (Eds.). Lecture Notes in Computer Science. Springer, Berlin. 256--272.
17. W. Keith Edwards, Mark W. Newman, Jana Sedivy, Trevor Smith, and Shahram Izadi. 2002. Challenge: Recombinant computing and the speakeasy approach. In Proceedings of the 8th Annual International Conference on Mobile Computing and Networking. ACM, 279--286.
18. Roger Evans, Paul Piwek, Lynne Cahill, and Neil Tipper. 2008. Natural language processing in C LIME, a multilingual legal advisory system. Nat. Lang. Eng. 14, 1 (2008), 101--132.
19. Giuseppe Ghiani, Fabio Paternò, Lucio Davide Spano, and Giuliano Pintori. 2016. An environment for end-user development of web mashups. Int. J. Hum.-Comput. Stud. 87 (2016), 38--64.
20. Hadley. 2009. Web application description language. Retrieved January 15, 2017 from https://www.w3.org/ Submission/wadl/.
21. Jan Humble et al. 2003. “Playing with the bits” user-configuration of ubiquitous domestic environments. In Proceedings of the Ubiquitous Computing (UbiComp’03). Springer, 256--263.
22. Philip N. Johnson-Laird. 1995. Mental models, deductive reasoning, and the brain. Cogn. Neurosci. 65 (1995), 999--1008.
23. Philip N. Johnson-Laird. 1983. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Harvard University Press.
24. Philip Nicholas and Johnson-Laird. 2006. How We Reason. Oxford University Press.
25. M. Cameron Jones and Elizabeth F. Churchill. 2009. Conversations in developer communities: A preliminary analysis of the Yahoo! Pipes community. In Proceedings of the 4th International Conference on Communities and Technologies. ACM, 195--204.
26. Steve Jones, Shona McInnes, and Mark S. Staveley. 1999. A graphical user interface for boolean query specification. Int. J. Digit. Libr. 2, 2--3 (1999), 207--223.
27. Vassilis-Javed Khan and Panos Markopoulos. 2009. Busy families’ awareness needs. Int. J. Hum.-Comput. Stud. 67, 2 (2009), 139--153.
28. Beata Beigman Klebanov. 2006. Measuring semantic relatedness using people and wordnet. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers (NAACL-Short’06). Association for Computational Linguistics, Stroudsburg, PA, 13--16.
29. Brian Y. Lim and Anind K. Dey. 2009. Assessing demand for intelligibility in context-aware applications. In Proceedings of the 11th International Conference on Ubiquitous Computing (UbiComp’09). ACM, New York, NY, 195--204. DOI:https://doi.org/10.1145/1620545.1620576
30. Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2119--2128.
31. Panos Markopoulos. 2016. Ambient intelligence: Vision, research, and life. J. Ambient Intell. Smart Environ. 8, 5 (January 2016), 491--499. DOI:https://doi.org/10.3233/AIS-160393
32. Panos Markopoulos. 2005. Designing ubiquitous computer-human interaction: The case of the connected family. In Future Interaction Design. Springer, 125--149.
33. Panos Markopoulos, Irene Mavrommati, and Achilles Kameas. 2004. End-user configuration of ambient intelligence environments: Feasibility from a user perspective. In Ambient Intelligence. Springer, Berlin, 243--254.
34. Irene Mavrommati, Achilles Kameas, and Panos Markopoulos. 2004. An editing tool that manages device associations in an in-home environment. Pers. Ubiquitous Comput. 8, 3--4 (2004), 255--263.
35. Nikolay Mehandjiev, Alistair Sutcliffe, and Darren Lee. 2006. Organizational view of end-user development. In Human-Computer Interaction Series. Henry Lieberman, Fabio Paternò, and Volker Wulf (Eds.), Springer, The Netherlands, 371--399.
36. G. Metaxas. 2010. End user programming of awareness systems: Addressing cognitive and social challenges for interaction with aware environments. PhD Thesis, Eindhoven University of Technology. DOI:http://dx.doi.org/10.6100/IR692128
37. Georgios Metaxas, Panos Markopoulos, and Emile Aarts. 2009. Amelie: A recombinant computing framework for ambient awareness. In Proceedings of the Ambient Intelligence. Manfred Tscheligi, et al. (Eds.), Lecture Notes in Computer Science, Springer, Berlin, 88--100.
38. Georgios Metaxas, Barbaros Metin, Jutta Schneider, Panos Markopoulos, and Boris de Ruyter. 2007. Daily activities diarist: Supporting aging in place with semantically enriched narratives. In Proceedings of the Human-Computer Interaction (INTERACT’07). Cécilia Baranauskas, Philippe Palanque, Julio Abascal, and Simone Diniz Junqueira Barbosa (Eds.), Lecture Notes in Computer Science, Springer, Berlin, 390--403.
39. Carman Neustaedter and Saul Greenberg. 2003. The design of a context-aware home media space for balancing privacy and awareness. In Proceedings of the Ubiquitous Computing (UbiComp’03). Springer, 297--314.
40. Thomas C. Ormerod. 2000. Mechanisms and strategies for rephrasing. Deductive Reason. Strateg. W. Schaeken, G. de Vooght, Vandierendonckt, and G. d’ Ydewalle (2000), Lawrence Erlbaum, 131--152.
41. John F. Pane and Brad A. Myers. 2000. Tabular and textual methods for selecting objects from a group. In Proceedings of the IEEE International Symposium on Visual Languages. IEEE, 157--164.
42. John F. Pane, Brad A. Myers, et al. 2001. Studying the language and structure in non-programmers’ solutions to programming problems. Int. J. Hum.-Comput. Stud. 54, 2 (2001), 237--264.
43. Siddharth Patwardhan and Ellen Riloff. 2007. Effective information extraction with semantic affinity patterns and relevant regions. In Proceedings of the EMNLP-CoNLL. 717--727.
44. Ted Pedersen, Siddharth Patwardhan, and Jason Michelizzi. 2004. WordNet: Similarity: Measuring the relatedness of concepts. In Proceedings of the Demonstration Papers at HLT-NAACL’04. Association for Computational Linguistics, 38--41.
45. Charith Perera, Arkady Zaslavsky, Peter Christen, and Dimitrios Georgakopoulos. 2014. Context aware computing for the internet of things: A survey. IEEE Commun. Surv. Tutor. 16, 1 (2014), 414--454.
46. Richard Power, Donia Scott, and Roger Evans. 1998. What you see is what you meant: Direct knowledge editing with natural language feedback. In Proceedings of the ECAI. 677--681.
47. Christopher Scaffidi, Mary Shaw, and Brad Myers. 2005. An approach for categorizing end user programmers to guide software engineering research. In Proceedings of the 1st Workshop on End-User Software Engineering (WEUSE’05). ACM, New York, NY, 1--5. DOI:https://doi.org/10.1145/1082983.1083232
48. ExistAlbrecht Schmidt. 2010. Ubiquitous computing: Are we there yet? Computer 43, 2 (2010), 95--97.
49. Vladimir M. Sloutsky and Yevgeniya Goldvarg. 2004. Mental representation of logical connectives. Q. J. Exp. Psychol. Sect. A 57, 4 (2004), 636--665.
50. Daniel Tetteroo et al. 2015. Lessons learnt from deploying an end-user development platform for physical rehabilitation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 4133--4142.
51. Khai N. Truong, Elaine M. Huang, and Gregory D. Abowd. 2004. CAMP: A magnetic poetry interface for end-user programming of capture applications for the home. In Proceedings of the Ubiquitous Computing (UbiComp’04). Springer, 143--160.
52. Jo Vermeulen, Geert Vanderhulst, Kris Luyten, and Karin Coninx. 2010. PervasiveCrystal: Asking and answering why and why not questions about pervasive computing applications. In Proceedings of the 6th International Conference on Intelligent Environments (IE’10). IEEE, 271--276.
53. Jin Yu, Boualem Benatallah, Fabio Casati, and Florian Daniel. 2008. Understanding mashup development. IEEE Internet Comput. 12, 5 (2008), 44--52.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Natural language interfaces

  • Ubiquitous and mobile computing