What are Natural User Interfaces?

It is already common to interact with a new class of devices entirely by using natural movements and gestures. The iPad, iPhone and iPod Touch, Xbox Kinect, Nintendo Wii, the new class of “smart TVs” and a growing list of other devices built with natural user interfaces accept input in the form of taps, swipes, and other ways of touching; hand and arm motions; body movement; and increasingly, natural language. These are the first in a growing array of alternative input devices that allow computers to recognize and interpret natural physical gestures as a means of control. Natural user interfaces allow users to engage in virtual activities with movements similar to what they would use in the real world, manipulating content intuitively. The idea of being able to have a completely natural interaction with your device is not new, but neither has its full potential been realized. What makes natural user interfaces especially interesting this year is the burgeoning high fidelity of systems that understand gestures, facial expressions, and their nuances, as well as the convergence of gesture-sensing technology with voice recognition, which allows users to interact in an almost natural fashion, with gesture, expression, and voice communicating their intentions to devices.

INSTRUCTIONS: Enter your responses to the questions below. This is most easily done by moving your cursor to the end of the last item and pressing RETURN to create a new bullet point. Please include URLs whenever you can (full URLs will automatically be turned into hyperlinks; please type them out rather than using the linking tools in the toolbar).

Please "sign" your contributions by marking with the code of 4 tildes (~) in a row so that we can follow up with you if we need additional information or leads to examples- this produces a signature when the page is updated, like this: - Sam Sam Jul 21, 2011

(1) How might this technology be relevant to the museums you know best?

  • Add your perspective here...
  • Another perspective here.

(2) What themes are missing from the above description that you think are important?

  • The note above captures some of the popular gesture and multitouch interfaces, but there's also a fairly broad categories of interfaces and interactions that take advantage of new kinds of sensing technologies — vision, ambient sensors, tangible object based interfaces, etc — which allow for natural interactions. That any object and any surface is the potential home for a virtual interaction and that people don't need to learn new interfaces are the real potential here. - bwyman bwyman Oct 13, 2012
  • - nhoneysett nhoneysett Oct 14, 2012 I don't think there is enough here about voice recognition's role in this HCI issue. Siri may be an over-hyped gadget, but voice recognition is more and more based on statistical translation - you might consider tying in these two technologies together. Its entirely possible that the accuracy of gestures will improve if they are based on statistical translation too.
  • Another perspective here.

(3) What do you see as the potential impact of this technology on education and interpretation in museums?

  • - - sheila.carey sheila.carey Oct 12, 2012 I think the potential is huge, especially for the youngest students who are introduced to computers not through the standard keyboard and mouse, but through tablets. There seems to be something intuitive about gesture-based interfaces.
  • These are the early stages of ubiquitous computing and is significant harbinger of things to come. It's down this path, that will allow rooms of touchscreens to disappear and that the virtual world of information can directly commingle with the physical world.- bwyman bwyman Oct 13, 2012
  • - nhoneysett nhoneysett Oct 14, 2012 Agreed. At some point, our kids will consider interacting with a computer using a mouse and keyboard, the way we look at interacting with a computer through a command line prompt. Can we include that star trek scene where Scotty picks up a present-day computer mouse and uses it as a microphone to talk to the computer?
  • - david.dean david.dean Oct 15, 2012 I believe both of the above views are correct. The Star Trek: Next Generation method of speaking to the computer and having both information and services occur upon request will become more the norm than fiction. SIRI is a step in that direction, and as voice recognition and background filtering technologies improve, such one-on-one interaction between the human brain and a vast network of cyber "brains" will become a reality. The impact of such a ubiquitous, consumer-driven method of cognition and processing will alter the very nature of museums' information delivery systems. What mustn't (and won't) be lost in this evolution will be the uniqueness of the "real thing," the collection object. What will change will be the way we, as museum professionals, talk about the objects with our visitors, through the devices they bring/wear.

(4) Do you have or know of a project working in this area?

Please share information about related projects in our Horizon Project sharing form.