A talk in Oslo "Things That Keep Us Busy - the elements of interaction"

I am very happy to be invited to give a talk at Oslo University next Friday, May 25th. The topic of the talk is my new book "Things That Keep Us Busy - the elements of interaction"

New course: The Elements of Interaction

This coming Fall I will teach a new (primarily graduate) course that I will develop over the summer. It is called "The Elements of Interaction" and it is based on my new book "Things That Keep Us Busy--the elements of interaction" (MIT Press 2017). The book is co-authored with my colleague Lars-Erik Janlert.

I am quite excited about this new course and I wonder if there are any courses like it out there. If you know, please let me know! And if you are interested or have questions, just email me.

Here is a description of the courses (at least as I plan now):

The Elements of Interaction

We are surrounded by interactive devices, artifacts, and systems. The general assumption is that interactivity is good -- that it is a positive feature associated with being modern, efficient, fast, flexible, and in control. Yet there is no very precise idea of what interaction is and what interactivity means. In this course, we will investigate the elements of interaction and how they can be defined and measured. We will focus on interaction with digital artifacts and systems but draw inspiration from the broader, everyday sense of the word. We will explore how the interface has changed over time, from a surface with knobs and dials to clickable symbols to gestures to the absence of anything visible. We will examine properties and qualities of designed artifacts and systems, primarily those that are open for manipulation by designers, considering such topics as complexity, clutter, control, and the emergence of an expressive-impressive style of interaction. The course is based on the idea that understanding some basic concepts and terms of interactivity and interaction can support interaction design and inform discussions about the future of interactivity.

The course is directed at graduate students with an interest in human-computer interaction in any form. Students in the course will develop a fundamental understanding of interaction and interactivity and an ability to analyze how interactive devices and systems interact and influence their users and environment.

The course will also help and support students to imagine new forms of interactions and to support them in the design and develop of interactive artifacts and systems.

The course will cover:

• The everyday reality of interaction and interactivity
• Understanding interaction and interactivity
• Basic elements and definitions of interaction
• How to study, analyze and measure interaction
• Examination of interaction complexity and control, interactivity clutter, etc.
• Implications for interaction design and UX
• The future of interaction (faceless interaction, interactivity fields, etc.)

As the core reading of the course we will use the book: 

“Things That Keep Us Busy – the elements of interaction” by Lars-Erik Janlert and Erik Stolterman (MIT Press, 2017)

A reflection after visiting CHI 2018 Montreal

Back home after a few days at CHI in Montreal. Overwhelming. Impressive in size, breadth and culture.

A lot can be said about HCI research, how it is changing, where it is moving, based on what happens at CHI. This is why I like to go to CHI. Not the individual papers or presentations, but the chance to get a sense of the whole field.

I made one new observation this year. I don't know if it is true or only a result of what I personally happened to listen to and see, but I will share it.

I experienced that the research presented at CHI seemed to drift and shift in two distinct ways. I saw more good design based research made by young design oriented researchers that take such an approach to research as given and as legitimate. I also saw more scientific research. People who study interactive technology and systems, but also different forms of interactivity, using traditional scientific methods and approaches. They did this not with the primary purpose of finding or developing new applications to real problems, that is, to support design, but with the purpose to develop a deeper understanding of certain aspects of human computer interactions.

I see these developments, both of them, as excellent for HCI research. For them to come together in one conference where they inform each other is the way it should be.

My personal critique of HCI research over the years has often been that it has been 'inbetween' research. It has not been real scientific research and it has not been real design oriented research. Being in between is the worst place to be as a researcher. It leads to confusion of purpose and of what the outcome is and how it can be evaluated. So, what I saw this year is positive and I hope it continues.

Studying the nature of interactivity (HCI)

I have always been drawn to research that is trying to understand the 'nature' of some phenomenon. There are several reasons for this. I am attracted to the idea that as a researcher I am actually improving my own understanding of the phenomenon I am studying and increasing my ability to explain its nature, and that this knowledge grows over time and is in some sense is cumulative. As a consequence of this, I have had two major research projects during my career. One has been to understand the nature of designing as a human approach to change and the other has been to understand the nature of the interaction between humans and interactive artifacts/systems.

I have realized that this aim is commonly confused with a radically different purpose which is to develop support that can improve peoples ability to design new technology and systems. As any kind of basic research, the results it produces will never be able to tell anyone what to design or create. Basic research primarily provides a deeper understanding of some phenomenon.  Instead of leading to prescriptive knowledge that could make design easier, it leads to increased complexity and from a design perspective often increased difficulty.

After being involved in basic research for many years I am more than ever convinced that it is not just necessary, it is also possible, even when it comes to a field like HCI. It is possible to study the nature of interactivity. It is possible to develop a deeper understanding of interactivity as a real phenomenon, something we, over time, can actually become better at explaining with sound and well-grounded theories and models.

For this to happen HCI research has to change some of its fundamental assumptions. For instance, HCI research has as a primary goal to lead to some form of (immediate) improvements or usefulness. In Habermas terms it is possible to say that HCI research is driven by a "control knowledge interest" and less of a knowledge interest of "understanding" or "emancipation" (which are his other two categories.  Of course, I am not saying that research with a "control" knowledge interest is not needed or should not be done. HCI is as a research field primarily aimed at improving practice so that is fine. But the field would benefit from also engaging in these other knowledge interests with a strong passion.

I am therefore both happy and proud of the work I have done with my colleague Lars-Erik Janlert about the nature of interactivity, where we did set out to do what I discussed above. We examined an existing phenomenon (interaction/interactivity) with the purpose to describe and explain its nature, to the best of our ability. Of course, we only reached so far. But I believe that it is a start that hopefully will inspire others. The result is the book "Things That Keep Us Busy -- the elements of interaction" (MIT Press, 2017)

One of the best books ever on design

Again I returned to the writings of David Pye. His writings have been with me since the early 80s. The nature and aesthetics of design" is one of the absolute best books ever written about design.

Interaction design, complexity, and virtuosity

One of the most preached principles in design, and particularly in interaction design, is to strive for simplicity. It is yet difficult to find any examinations of what simple really means when it comes to design (there are some good exceptions, such as Maeda and Mollerup, see references below).

In many cases, being simple is of course good. Nevertheless, we also know that we live in a world that is complex and sometimes requires complex actions. We also know that people can do amazing things even with devices that are highly complex. Virtuosity can be achieved. So, the question becomes, can we design artifacts that require complex actions in a way that could support the efforts of reaching virtuosity?

Below is an excerpt from our book "Things That Keep Us Busy - the elements of interaction" (MIT Press, 2018). This is from Chapter 6 "Control".


"6.5 Virtuosity

Can we imagine artifacts that are highly complex while still being inviting to a user and providing incentives for continuous engagement, maybe even spurring a few users to aim for extreme levels of mastery? Let us consider the violin example again. As we noted in chapter 5, the violin combines low internal and external complexity with high interaction complexity, which apparently invites a range of user behaviors. Most people initially find the violin extraordinarily difficult to interact with even though the artifact itself is quite simple: just a few strings mounted on a soundboard and a bow. Beginners are not able to make any real music on the violin even though they of course can make (terrible) sounds.

We also know that the violin invites virtuosity, a display of expertly handled extreme interaction complexity. Virtuosic violin performances have a long tradition and many famous composers have written music specially tailored to let distinguished players show off their virtuosity. When it comes to musical virtuosi, it is not uncommon to hear comments that virtuosity is more a technical achievement or circus act than an expression of musical insight and depth of interpretation. It may be that listeners, rather than being moved by the music, are impressed and awed by the display of almost superhuman skills.

Why is it that some artifacts seem to invite virtuosic use while others don’t? Are there examples of digital artifacts or systems that have the same virtuosity-inviting quality as the violin? There seem to be few parallels to the violin example: low external complexity yet still inviting virtuosity. We have to remember that up until recently digital artifacts and systems have typically had relatively high external complexity. The development of human–computer interaction has been dominated by a constant effort to refine and keep controllability on par with ever-expanding functionality, rather than any ambition to lower external complexity, not surprisingly resulting in externally complex artifacts and systems. Of course, lately with the proliferation of small digital artifacts and apps with specialized functionalities, increasingly under the pressure of the interface bottleneck
problem, external simplicity has become a key issue and mark of good design. However, these artifacts do not seem to invite virtuosity, perhaps because the interaction complexity usually is low, too low.

Another explanation of why we have not seen convincing examples of virtuosity with digital artifacts could be the up till now typically strongly discretized input and output and strict turn taking of digital artifacts, quite far from the analog and continuous flow of violin playing. However, discrete input combinations related to desired output combinations in a complex manner do exist, for instance in some computer games, which open for the possibility of some sort of virtuoso performances; similarly, computer hackers can make dazzling performances of rapidly finding and fixing software problems, hammering away at breakneck speed on a command-based interface. Even with more mundane examples, such as highly complex office software with huge numbers of commands, layers of functionality, we may be impressed by the brilliant technique the professional user displays. Such examples demonstrate fast decision making under pressure in a situation where there is a lot going on to keep track of. This could potentially be understood, and savored, as a kind of virtuosic performance.
A difference is that the interaction is overtly digital and the external complexity considerable: it is a form of “combinatory” virtuosity rather than the “smooth” virtuosity of top-class violin playing. A violin seems to allow more room for users to express themselves in a way that goes beyond functional achievements, possibly that might have something to do with the smoothness and very fine nuances the instrument affords the player. But the distinction between analog and digital interaction, from the user point of view, has become very blurred by now; first with the advent of graphical user interfaces (GUI) and pointing devices, then with new tracking, sensing, and presentation techniques partly deriving from virtual- reality technology, and with tangible user interfaces (TUI) where physical objects are used in analog mode to interact with the digital artifact, and lately with the breakthrough of gestural interaction. The stage is now set for new applications and forms of interactions that could be much more like violin playing. In fact, we already have some artifacts and systems like these around, for instance, in the form of games that with quite simple interfaces invite and compel the user to develop sophisticated interaction skills, often virtual versions of existing “real” games and sports. Using a gesture-controlled golf simulator seems to be close enough to the “real” thing to let expert golfers show their virtuosity in the simulator.

Although we have focused on the violin as a possible model and inspiration for future artifacts and systems inviting virtuosity, we do not want to exclude designs with high external complexity from consideration in this respect. After all, the piano and other keyboard instruments, externally much more complex than the violin, also often figure in virtuosic compositions and performances. Still, the external simplicity of the violin by its very sharp contrast to older-style digital artifacts and systems perhaps makes it a more intriguing and challenging model for a designer. We believe that virtuosity as a manner of interaction could, and maybe even should, be reconsidered and revisited in interaction design. There may be situations and technology use that would benefit from such a perspective. The history of virtuosity is rich, ranges over many fields, and might provide us with new insights into interaction design possibilities with contemporary technology.

There are of course arguments against a move toward virtuosity in the field of interaction design. One obstacle to virtuosic interaction with many digital artifacts is their short market lives and rapid development: the violin has been around for hundreds of years with hardly noticeable changes, giving the community of users ample opportunities to develop a culture of virtuosic use. Many of the new digital artifacts, in contrast, are on the market less than five years before they disappear or are superseded by completely different designs, sometimes building on new and different technologies. Another argument against virtuosity is of course that it requires extensive training over a long time period. We admire virtuosity since most of us do not have the time and maybe talent to engage in training to the degree needed. So, to require virtuosity from users may restrict the group of potential users to almost none.

A more obvious factor working against virtuosity is that while it might be a sometimes-desired manner of interaction, such extreme interaction is not typically what designers and users look for. In much everyday artifact use, most users care for little more than very basic performance—and there often is a conflict between the requirements of high-level performers and low-level performers that prefer low interaction complexity. But, maybe a general change of attitude and the demands of professional users can change that; we have actually come a long way from the time when “user- friendliness” was the key criteria, and we still keep moving."

---------------------------
Maeda, J. 2006. The Laws of Simplicity. Cambridge, MA: MIT Press.
Mollerup, P. 2015. Simplicity—A Matter of Design. Amsterdam: BIS Publishers.

VR, authenticity and killer apps

The hype around VR has lately been growing and ads are making the case that it is time to take on this new form of interaction for all kinds of applications. However, in a great article (published by TechCrunc) the writer  Sibjeet Mahapatra argues that there is a major problem in the world of VR. The problem is the missing killer app.

Mahapatra discusses what he sees as the two values that VR can offer, primarily a sense of presence. But he also argues that VR still lacks when it comes to authenticity. The author makes a good case for what is needed for VR to really become something wanted by a larger audience. So, a good read about interaction and interactivity.

[And of course, I always like someone who  mentions Rorty and his "experience machine"!]