Defining the Intelligent part of Intelligent User Interfaces

Since I’m talking about “AI” here, and took the time to define IUI, I figured it was probably worth some time offering my thoughts on AI…

Let me start off with a disclaimer: I’m not a machine learning engineer. I’ve never written a line of production AI code… I say production code on purpose, because the truth is, I have written some code that I consider to be “smart”, but more as a hobby. I actually began my career back in 1998 as a software engineer and took an interest in AI quite some time ago. I moved from software development to design about 5 years later and haven’t written production software since about 2003.

I first started digging into AI back in the early 2000’s, and the book that served as my introduction to AI was ‘Constructing Intelligent Agents using Java’. The principles that I learned from reading that, and playing around with some of the sample applications really became the foundation of my understanding of AI. I’ve built a few little things along the way, mostly just hobby projects – I built a little SPAM detection app and a new reading app that grouped similar stories (so I didn’t have to see 10 different versions of the same “Apple announces new iPhone”).

As someone who conceives and designs products for a living, I think it’s really helpful to have a solid understanding of what is possible with technology, so I’ve made sure to stay up to date with what is happening in the field as much as possible.

The way that I think about it is, there are really two kinds of “AI”, sometimes referred to as Narrow AI and Strong AI.

Narrow AI applications are optimized for a single problem or domain. As an example, there is a company that makes a system called Smart Airport, the domain is Airport Logistics – this is narrow AI aimed at optimizing the flow of an airport (planes, people, etc.). Just think about the domino effect of one delayed plane, a bad snowstorm, or mechanical issues with a plane. A smart system can run through all the scenarios of how to recover from that much faster than humans. Narrow AI can be incredibly smart, however, if you tried to use a system tuned to the operational efficiency of an airport to run the logistics of a different domain, say a sports venue, it would probably fail miserably.

Strong AI, on the other hand is what most people think of when they talk about AI: “human like intelligence”. As an example, we weren’t born with the knowledge of how to use the Internet, but we learned how to use it. That was not “programmed in”. Nor were we born with the knowledge of how do design things, we learned that. Strong AI, or “General Intelligence” is best characterized by its ability to learn to operate in any domain.

Recently there has been a lot of talk about Deep Learning, and the way that I like to think about this is that it falls somewhere between the two.

DeepMind, in case you haven’t heard of them, is an Artificial Intelligence company that Google acquired. They developed a Deep Learning algorithm that actually learned how to play – and win – video games. It started with some of the classic arcade style video games many of us grew up with, like Pac Man. Okay, you’re probably thinking, so? Who cares? Well, what made it so astonishing is that they didn’t teach the system the rules of the games, they just let it play the game until it learned the rules on it’s own. One of the games it learned to play was Boxing, and not only did it learn how to win, it learned how to optimize winning. It found, on it’s own, that you could pin the opponent in a corner and run up the score. At the time Google acquired them, it had mastered about 2/3rds of the 35 or so games that is was learning to play.

Fast forward to 2015 and DeepMind accomplished what most people thought was an impossible task for Artificial Intelligence, it beat a human champion at Go – in case you’re not familiar with it, Go is an ancient Chinese game in which you place stones on a 19 by 19 board, and capture your opponent’s stones by surrounding them. The rules are very simple, but they give rise to a complex, subtle game.

There are a number of articles online that describe why this accomplishment is such a big deal, but the simple explanation is that unlike Chess, which was solved using Brute Force algorithms, Go is different. Every single move in Go gives rise to infinitely more possible responses. If the average possible response to a move in chess could be one of about 35 moves, in Go the “Branching Factor” is about 250. To give you a sense of what that means, if you want to think 2 moves ahead in chess, there are about 1235 moves to consider (35 x 35). In Go, it is about 62,500 (250 x 250), and three moves would be 15,625,000 (250 x 250 x 250).

Winning at go requires a sort of an intuition of the board, something that a champion develops. This kind of intuition is something that Brute Force algorithms just can’t deal with.

Based on what I wrote above, you’d think that Deep Mind was Strong AI, because it learned on it’s own and made decisions to achieve it’s goals, and that is correct. However, the question of whether Deep Learning is really Strong AI is whether or not it could function in other domains? Could the same Deep Learning algorithm that learned to play & win those video games also run a supply chain? I’m not sure, my suspicion is that it couldn’t, therefore, my guess is that it falls somewhere between Narrow & Strong (but honestly, I’m not really smart enough to know for sure).

Just to be complete here, there is actually another category, which is called Super Intelligence, which is really a natural evolution of General Intelligence. If AGI can learn, then why can’t it learn to improve itself? This is what all the commotion has been about over the last couple of years with people like Elon Musk & others warning about our impending AI Doom.

Back in 2014, I had the good fortune of attending ICRA, the International Conference on Robotics & Automation. One of the workshops I attended was the Workshop on General Intelligence for Humanoid Robots. The guy who organized that is a pioneer in the field of General Intelligence, Ben Goertzel. During his presentation he said something that has really stuck with me:

…so, AI includes, conceptually, making systems that are intelligent like C3P0, HAL9000 or a thousand times smarter than any human being. The field of AI also includes “Expert Systems“ for say, medical diagnosis, that just goes through a hand coded list of rules or say a neural net control system for a self-driving car, which is highly specialized for that type of car. AI is a very big umbrella, so it’s not particularly clear where AI leaves off and algorithms begin, is there really a big difference between all the algorithms in an AI textbook and the algorithms in an algorithms textbook? Drawing borders between disciplines is not the most interesting thing, the world cuts across all the disciplinary boundaries anyway.

Ben Goertzel

That notion of it’s all just algorithms has always been something that I’ve kept in mind when talking about “AI”.

At the end of the day, for the purposes of this blog, I’m going to consider something intelligent as long as it as it loosely conforms to the definition I laid out in my first post here (What is IUI?)

Improving the acumen, acuity, and productivity of people by applying computational intelligence to experience design.

What do you think?


What is IUI?

IUI is an acronym for ‘Intelligent User Interfaces’. This is not really anything new. As a concept, it’s been around for more than 20 years, but given all the recent advances in Machine Learning / A.I. it’s become increasingly possible to improve the experiences we design for people by applying some computational intelligence.

I think the name says it all, User Interfaces that are intelligent, but I wanted to see if I could find a really good definition. One of the first results in Google comes to us from Wolfgang Wahlster, who is Computer Science Professor in Germany. He defines IUI as:

Intelligent user interfaces (IUIs) are human-machine interfaces that aim to improve the efficiency, effectiveness, and naturalness of human-machine interaction by representing, reasoning, and acting on models of the user, domain, task, discourse, and media (e.g., graphics, natural language, gesture).

Wolfgang Wahlster

I really like several parts of that:

  1. He starts with calling out Human-machine interfaces. We’re building something for humans, for people. Love that he starts with people first!! That is the right orientation! I like the word Human better than User, also.
  2. He identifies goals for IUI – to improve the efficiency, effectiveness… of interactions. Those seem like good goals to me! We’re trying to make life better for people.
  3. Finally, he calls out how to achieve it… reasoning, and acting on models of the user, domain, task, etc. We’re going to have the computer do some work for people. Cool!

One of the other top results from Google is the IUI page on Wikipedia. I was a little less impressed by this explanation of IUI, but it’s simple:

An intelligent user interface is a user interface that involves some aspect of artificial intelligence (AI or computational intelligence)…


What made the Wikipedia explanation so interesting was that they went on to include an example of IUI, and the example they cited, was… Clippy!

…there are many modern examples of IUIs, the most famous (or infamous) being the Microsoft Office Assistant, whose most recognizable agentive representation was called “Clippy”


If you aren’t old enough to recall clippy, I’ll put a link at the bottom of this post to a good article on it. Basically it was a little animated paper clip that appeared when you used Microsoft Office Applications and he would offer to help you do things.

Let me hit pause on Clippy for a second…

One of the reasons I wanted to get a good definition for IUI is that being able to define it enables us to determine if something is actually IUI or not.

If you look at how the Wikipedia article defines IUI, then yes, Clippy is an example of it. However, I’m not sure that it really made life more effective or efficient for people, as Professor Wahlster calls out. Regardless of how successful Clippy was, though, it was an interesting idea to try and with all the amazing advances in Natural Language Processing, we have today, a modern version of Clippy could be pretty powerful in helping people writing more compelling & influential documents, as one example. There are plenty of companies that are in the Natural Language Generation business (like Narrative Science, Automated Insight, etc) so we know machines are capable of writing like humans and that capability is only going to get better over time.

Returning to our definition… we have a basic overview of IUI: it’s about making life easier / better for people by applying AI to user interfaces. We’ll go ahead and offer a definition of IUI as:

Improving the acumen, acuity, and productivity of people by applying computational intelligence to experience design.

There is a great quote by William James that I think of when the subject of AI comes up:

The more of the details of our daily life we can hand over to the effortless custody of automation, the more our higher powers of mind will be set free for their own proper work.

William James

There was a bit more to that quote, and his point was really about creating habits to reduce our daily cognitive load… but I love the quote nonetheless and think it fits here pretty well. This is really one of the promises of IUI, to reduce cognitive load when people are using technology.

On the off chance that you’re new to IUI, there is actually an annual ACM Conference on the Topic of IUI, and it’s entering its 24th year!

The tagline for the conference is GREAT: “Where HCI meets A.I.”

OMG! That is great! Instead of stealing it, which I thought seriously about, I’m going to tweak it a bit… my tagline is: Improving HCI by Applying AI.

As with most ACM conferences, it’s fairly technical and academic, so I don’t recommend it for most people. If you aren’t able to make it, no worries, I’ll be providing some great coverage of the event here. If you are going, let me know, I’d love to meet you!

In addition to covering conferences, like IUI, some other things I’ll be covering here will include examples of IUI in the real world, ideas I have, design principle, IUI Patterns, interviews and some general articles of interest. Hope you find it useful!

If you have question, a topic you want covered, or an example of IUI that you’d like to share, I’d love to hear about it!