IUI Example – Creative Inspiration, from Google

If you read any of “the year ahead” predictions for 2019, or even ones from the last few years, one thing you’ll undoubtedly come across is that Robots or AI will eventually take everyone’s job… maybe not today, or tomorrow, but eventually. I personally don’t buy this line of thinking, and think that Marc Andreessen had it ‘mostly’ right in a post he wrote way back in 2014: “This is probably a good time to say that I don’t believe that robots will eat all the jobs…”

One of the most interesting topics in modern times is the “robots eat all the jobs” thesis. It boils down to this: Computers can increasingly substitute for human labor, thus displacing jobs and creating unemployment. Your job, and every job, goes to a machine.

This sort of thinking is textbook Luddism, relying on a “lump-of-labor” fallacy – the idea that there is a fixed amount of work to be done…The counterargument to a finite supply of work comes from economist Milton Friedman — Human wants and needs are infinite, which means there is always more to do. I would argue that 200 years of recent history confirms Friedman’s point of view.

Marc Andreessen

Marc then goes on to rip apart the Robots take all the jobseventuality” with a number of really compelling arguments and thought experiments.

The post is incredible and I’m probably not smart enough to do it justice, but let me try to offer a quick summary of his arguments and then I’ll provide my take… and circle back to the creative inspiration title of this post!

The overly simplistic version of the points in his very long post are:

  1. For the Luddites to be right today (that robots / AI will take all the jobs) one has to believe that there won’t be any new wants or needs (which runs counter to human nature) and that people / humans won’t contribute to those things being created
  2. While it’s true that automation / technology displaces work (and that is something that must be addressed), the flip side is that the same automation / technology increases standards of living. The way that I’ve always thought about this is thinking about all the safety features in cars – blind-spot monitoring systems were only available in high end vehicles a few years ago and now most cars have them, this is a life-saving feature… not just a cheaper big screen TV. Technology enables both of these…
  3. He offers three suggestions to help people who are hurt by technological change: first, focus on increasing access to education. Second, let markets work. Third, create & sustain a vigorous social safety net… with those three things in place, humans will do what they’ve always done: create things to address and/or create new wants and needs.

Now, I love all three of those, and it would probably be enough to stop there… but Marc goes MUCH further…

  • The flip side of robots eating jobs is that this same technological advancement also democratizes manufacturing – it puts the power of production in everyone’s hands! I love this thinking!
  • Costs go down / things get cheaper… robots producing things means lower costs, which lead to falling prices, which stretches people’s purchasing power and raises people’s standard of living.

He wraps up the long post by restating his “this is a good time to state that I don’t think robots will eat all the jobs…” and offers the following…  which I’ll take one by one…

First, robots and AI are not nearly as powerful and sophisticated as I think people are starting to fear. Really. With my venture capital and technologist hat on I wish they were, but they’re not. There are enormous gaps between what we want them to do, and what they can do.

Marc Andreessen

I think Marc is still right on the first one. I’m a goofy optimist. Always have been, always will be. There is a line that someone used on me once that I’ll never forget: “don’t mistake a clear vision for short distance” and while technically true given the subject we were discussing, one never knows where the next breakthrough is going to come from! Again, he was right in 2014, but probably less so every day.

One other point I’d like to make here is that they don’t have to be as powerful & sophisticated as people talk about in the general press. What I’m focused on are the specific examples of real progress. I think we are a long way from the “Singularity” (great video here) but trying to pinpoint it’s arrival isn’t really super exciting to me, not practically anyway.

The questions to really be asking are: “which jobs”, “when”, and “how”. My thought is that it won’t be some zero-sum game.

Second, even when robots and AI are far more powerful, there will still be many things that people can do that robots and AI can’t. For example: creativity, innovation, exploration, art, science, entertainment, and caring for others. We have no idea how to make machines do these.

Okay, this is one of the main reasons I wanted to write about this, and as someone who works in a creative field, Creativity is something that I’m incredibly interested in… so I was a bit floored when I started digging into Magenta…

A primary goal of the Magenta project is to demonstrate that machine learning can be used to enable and enhance the creative potential of all people.

There is a lot to Magenta, so I’ll focus on two parts that really caught my attention.

First, is the Sketch-RNN – given a source sketch, it will auto generate additional sketches. Pretty rudimentary, but two things: first, it is a rudimentary starting image and second, imagine what this could do over time?

Second is Music Transformer – given a starting sequence, it will generate music with long term coherence to the original sample provided.

It doesn’t take much imagination to come up with ideas on where were going to start seeing uses for this type of innovation. If you’ve ever played with Garageband on your iOS devices (or the full blown Logic Studio on a Mac, or similar DAW software), you can start to think that we’re seeing a further advancement in the democratization for the creation of art & entertainment.

As designers, the sketching stuff should be of particular interest. One of the most important parts of my design process is the Diverge / Emerge / Converge diamond. I can see a future where tools like this will help us quickly explore more divergent ideas.

You should really go check out the samples on that page, they are truly incredible.

And check out some of the other demo apps.

I thought about ending here, but there are two more points to cover, so back to Marc…

Third, when automation is abundant and cheap, human experiences become rare and valuable. It flows from our nature as human beings. We see it all around us. The price of recorded music goes to zero, and the live music touring business explodes. The price of run-of-the-mill drip coffee drops, and the market for handmade gourmet coffee grows. You see this effect throughout luxury goods markets — handmade high-end clothes.

Marc Andreessen

I love this line of thinking. Imagine a future where we see labels in clothes that read “Made by Humans” like the “Made in the USA” labels we see today?

Finally, his last point…

Fourth, just as most of us today have jobs that weren’t even invented 100 years ago, the same will be true 100 years from now.

Marc Andreessen

That final point is so true. Marc actually ends his post stating he is ‘way long’ on human creativity, as am I…

What do you think? Will technology take our creative jobs away? Change them? Thoughts?


IUI Example – eBay

I wanted to call this post ‘Reducing Decision Fatigue’, but the reality is that most of the posts I’ve written here could have that title! 🙂 As cited in my recent post about Design Principles, I think a core principle of IUI is to help people make smart decisions quickly.

One of the great papers at the 2017 AAAI (Association for the Advancement of Artificial Intelligence) Spring Symposium was ‘Communicating Machine Learned Choices to E-Commerce Users’. It was written by a bunch of folks at eBay… and the basic premise was that you could use Machine Learning to help guide people through a long list of products by grouping them based on attributes (new vs. used, seller rating, etc.) that were most relevant to the purchase decision of a given product… but doing so required making good design decisions.

The abstract:

When a shopper researches a product on eBay’s marketplace, the number of options available often overwhelms their capacity to evaluate and confidently decide which item to purchase. To simplify the user experience and to maximize the value our marketplace delivers to shoppers, we have machine learned filters—each expressed as a set of attributes and values—that we hypothesize will help frame and advance their purchase journey. The introduction of filters to simplify and shortcut the customer decision journey presents challenges with the UX design. In this paper we share findings from the user experience research and how we integrated research findings into our models.

They started by analyzing historical transactions to identify inherent value placed on specific attributes, and identified them as “global” or “local”. Global attributes are ones that are common across products (e.g. condition) and local attributes are ones that are specific to a subset of products (e.g the OS version of an Android phone), and some of the local attributes actually replace the global attributes (e.g. ‘Rating’ for baseball cars replaces ‘Condition’)

They then came up with something they called the ‘Relative Value’ of an attribute, which basically looked at the premium that shoppers paid for a product given the value of that attribute (e.g. a returnable item vs a non-returnable item).

In the above image, we see that the higher price paid when an item is returnable.

They then went on to review Behavioral Signals, to determine which attributes were “Sticky” and which attributes were “Impulsive” during a shoppers decision making process. Sticky attributes are obviously ones where buyers stick to a specific value (or range) in their purchase journey significantly more than random chance would dictate. Impulsive attributes are ones that correlated with impulsive transactions (short view trail before purchase).

Once they identified the attributes that really mattered, it then came time to figure out how to design the experience… and there were three parts that they covered:

  • Filter Naming – how to communicate understandable and compelling filter titles?
  • Filter Overlap – how to communicate that filters are not mutually exclusive
  • Filter Heterogeneity – how to communicate why eBay is displaying unrelated filter sets in close proximity

For the filter naming, each one could include one or more attributes (global or local) and they were constrained by the need to identify each of the filters with a human readable name. For example, for products where people prefer buying things that are new and want to have the flexibility of returns, and are weary of overseas shipping, they had a theme named ‘Hassle Free’.

Then the Usability testing began, where they tested a variety of titles – from “emotive & engaging” to “simple & descriptive” They discovered a few things:

  1. People overwhelmingly preferred simple titles.
  2. Item condition was the first reference frame most people locked into
  3. People found longer titles, especially those with compound filters, were difficult to understand

They landed on B, the descriptive titles split over two lines.

One of the Design Principles that I recently wrote about was ‘Developing Trust’, so it was really cool to see the following:

User study participants also expressed low confidence in our recommendations when the inventory covered using ML filters was smaller than that of search results. For example, when the value based filters are concentrated on one or two attributes, significant inventory may be left out. We addressed this concern by taking inventory coverage into consideration in our ML research.

They then go on to say…

We also added navigation links for shoppers to explore the entire inventory beyond our recommendation, which has helped us gain users trust in our recommendations. These links to “see all inventory” also provide easy access to listings not highlighted by our filter-sort formula, in support of cases where a shopper’s ‘version of perfect’ went undetected by our analysis.

This is such a cool example of leveraging machine learning to help people make decisions.

What do you think?

IUI Example – Google Flights

IUI Example – Google Flights

Google just rolled out a new feature to Google Flights, that is pretty cool – they are now predicting if your flight is going to be delayed.  As I’ve mentioned previously, I love to travel and really dig any innovation that will improve my travel experience.

As I read about this, I couldn’t help but recall a presentation I saw by Aparna Chennapragada, VP of Product Google, during an O’Reilly AI conference.

All systems imperfect — there will be a precision / recall tradeoff in almost any system that you rely on. But what you want to pay attention to, as a practitioner, is the cost of getting it wrong. Let me give you an example. Let’s say that you’re building a search system and you return a slightly less relevant article in a search result… it’s not the end of the world. But then let’s say that you build a local search product, where you inform the person searching that, yes, Home Depot is open, you should go now. Then, the person gets in the car, goes to Home Depot, and it’s closed, and they say “What the Hell?”. The cost of doing that, the cost of getting that wrong is higher.

She then gives the example of when they were building Google Assistant…

When we were working on the Google Assistant, and we say, hey, you’re flight is on time, you can leave right now and it takes 45 minutes to get to the airport and then you go to the airport and you miss the damn flight and can’t speak at the conference, then the “What the Hell” is much higher.

There are a number reasons a flight can be delayed or cancelled:

  • Mechanical Issue with the plane
  • Weather (at both the departure as well as the destination airport)
  • Late inbound aircraft
  • Crew
  • Etc.

What Google seems to be doing is simply tracking the inbound aircraft, either by gate numbers – if a flight to say New York  is departing at supposed to depart at 8:21 PM and the incoming flight to that gate is delayed, there is a great chance that the New York flight will be delayed. I’m sure they are doing more than that, they probably have tons of historical data and some good algorithms that take things a little further.

As a side note, each one of these is well known, and airlines have operational departments to deal with issues as they arise. I even read a great book a few years ago – ‘A New Approach to Disruption Management For Airline Operations Control’ that went into detail about a proposed multi-agent, intelligent system to improve operations. I also talked about Smart Airport system in a recent post.

The big takeaway here is that when you’re building things like this, it’s really critical to understand the costs of being wrong and what it means to the person using it!

What do you think?

IUI Example – Kayak

This post is incredibly personal, I love to travel. Just look at my profile on Twitter…

Anyway… 🙂

One of the biggest questions someone has when they’re looking to book a flight is whether the price will go up or down, in other words, should they buy now or wait?

Kayak offers people an answer to this question with a little indicator “OUR ADVICE”.


If you read my recent post on IUI Design Principles, the very first one was “Raise People’s Acumen”:

Acumen is roughly defined as the ability to make good decisions, quickly. Where a principle like this works really well are for things like analytical tools. As an example, if you’re designing a dashboard, think about the decisions that someone would make with the data and figure out how you can enable them to make better decisions, faster. Another way that I’ve written this principle is “Help people make smart decisions quickly”.

This is so perfect. They are answering that critical question of whether or not to buy now.

But they don’t stop there. They also follow one of my other main principles when building Intelligent User Interfaces, “Be transparent, the real job is developing trust

If a machine is going to do something or make a suggestion for a person, they should have the ability to see how that output was chosen. Look for ways to provide some transparency in the system that gives people trust & confidence.

They put a little ‘i’ icon that people can click to provide some detail behind the advice

This is so brilliant.

I’m not sure the explanation is quite as robust as it could be, however…

Let me explain…

This incredible little innovation didn’t originate at Kayak. The company that invented this was actually called Farecast, which Microsoft acquired in 2008… and, shockingly, they don’t offer this when you search for flights on their site.

One of the reasons that I know that the explanation could be better is because I know some of the history of Farecast.

Farecast was founded by Oren Etzioni, a computer science professor at the University of Washington. He came up with the idea back in 2002 when he was on a flight and learned that the people sitting next to him paid much less for their tickets simply by waiting to buy them until a later date. So he had a student go try to forecast whether particular airline fares would increase or decrease as you got closer to the travel date. With just a little bit of data, the student was able to make pretty accurate price predictions on whether someone should buy or wait.

From there Etzioni built Farecast. It was just like other online airfare search sites (OTAs), with one major addition: it added an arrow that simply pointed up or down, indicating which direction fares are headed.

The company, which was originally named Hamlet and had the motto of “to buy or not to buy”, was built using 50 Billion prices that it bought from ITA Software (which was acquired by Google in 2010). ITA is a company that sells price information to airlines, websites, and travel agents, and has information for most of the major carriers. When Farecast bought the data from ITA, it didn’t have prices for JetBlue or Southwest, but could indirectly predict fares for those carriers based on fares from the carriers it did have pricing data for.

Farecast based it’s predictions on 115 indicators that were reweighed every day for every market. They paid attention not just to historical pricing patterns, but also included a number of other factors that shifted the the demand or supply of tickets – things like the price of fuel, the weather, and non-recurring events like sports championships… anyone buy their tickets for Qatar 2022 yet? 🙂

As mentioned at the start of this post, I love travel… and there are some other travel examples I’ll be sharing in upcoming posts (including one from Google Flights).

What do you think? Any examples you can think of leveraging IUI for travel??

IUI Example – AirBnB

One of the principles that I covered in my recent post on IUI Design Principles, was: “Reduce Cognitive Load”.

Part of the inspiration for that principle was an article I read a few years ago about a pricing tool that AirBnB built for people listing their properties.

The problem, they had discovered, was that critical moment when a person is trying to do decide what price to charge when they’re listing their property:

In focus groups, we watched people go through the process of listing their properties on our site—and get stumped when they came to the price field. Many would take a look at what their neighbors were charging and pick a comparable price; this involved opening a lot of tabs in their browsers and figuring out which listings were similar to theirs[clip] …some people, unfortunately, just gave up

Dan Hill, Product Lead @ AirBnB

There is really so much to love about the discovery of that insight. At the risk of stating the totally obvious, if people don’t list their properties, the supply side of the market starts to erode, which isn’t good. 🙂

AirBnB isn’t the only company that has to deal with pricing, obviously! I’ve spent a lot of the last 20 years in the Consumer Goods / Retail domains, and think about the number of products on the shelf at your local grocery store, those prices were set by a person.

The awesome thing is how they solved this, and it is a great example of a truly intelligent interface! They actually built a tool called Aerosolve, a machine learning algorithm to provide intelligent pricing recommendations to people listing their properties.

They aren’t the only company that has used algorithms to set prices. Think about ride sharing apps like Uber, Grab or Go-Jek, prices change based on different variables: distance, weather, and demand (give or take). But those are pretty simple, all known quantities and over time develop a tremendous amount of historical data… e.g.  how much more are people willing to pay when it’s raining? In other words, price elasticity.

Setting prices for a property on a site like AirBnB is a little more complicated. In a technical paper that Dan Hill wrote (which is where I got the quote above), he covers both some of the history or Aerosolve as well as some of the challenges in building it.

In addition to all the normal things you’d expect to be factors, like number of rooms, WiFi, seasonality, and so on, there are some other interesting factors that come into play. It turns out that the number of reviews plays a large part in pricing, people are willing to pay more for a listing with good reviews (seem obvious retrospectively).

But what about big events?

Source AirBnB

Consider SXSW, as depicted in the above graphic, what about the World Cup? Those are obvious big examples, but there are events in cities all the time that don’t get the press of things like those events? AirBnB has to account for those as well.

It’s interesting to note here the origin story of AirBnB was that Brian Chesky came up with the idea for the site when we wanted to attend a design conference, realized that all the hotels in SF were sold out, and decided he could pay for his ticket to the conference by renting an air mattress in his apartment to someone who wanted to come to SF but couldn’t get a hotel room.

One of the other really interesting things is the way they had to deal with geographic boundaries for properties. An early version of their algorithm simply drew a circle around a property, and considered anything within that radius a “similar listing”, but what they discovered was that simplistic view had a serious flaw…

Imagine our apartment in Paris for a minute. If the listing is centrally located, say, right by the Pont Neuf just down from the Louvre and Jardin des Tuileries, then our expanding circle quickly begins to encompass very different neighborhoods on opposite sides of the river. In Paris, though both sides of the Seine are safe, people will pay quite different amounts to stay in locations just a hundred meters apart. In other cities there’s an even sharper divide. In London, for instance, prices in the desirable Greenwich area can be more than twice as much as those near the London docks right across the Thames.

We therefore got a cartographer to map the boundaries of every neighborhood in our top cities all over the world. This information created extremely accurate and relevant geospatial definitions we could use to accurately cluster listings around major geographical and structural features like rivers, highways, and transportation connections.

Dan Hill

What a great example of improving the experience people have using a tool by applying computational intelligence.

One of the things I’m going to cover in an upcoming post is how to discover opportunities for IUI.

In the meantime, I’d love to hear what you think about what AirBnB are doing! Thoughts?

IUI Example – Intelligent Remote Control

Picture the remote control from your TV / Cable / Satellite provider.

Chances are, the image in your head is similar to the image most other people come up with. We all know what a remote control looks like. There are rows and rows of buttons, some bigger than others, some with alphanumeric characters, some with symbols, some round, and some rectangular. There is something for power, volume, changing the channels, and a whole host of functions that you probably use very infrequently. Remote controls haven’t changed much in years… they are what they are.

I really began thinking about remote controls back in 2011 after reading the really good book Simple & Usable by Giles Colborne.

In the book he outlines Four strategies for simplicity, and he does so by describing an interview exercise that he runs job applicants through. What he does is ask them to “simplify” the remote control for a DVD player.

Back in 2011, most people probably still had a DVD player, and this exercise presents some tricky problems.

Typically, a DVD remote has about forty buttons, many have more than fifty, and, as GIles suggests, that seems excessive for a device that is used to play and pause movies. When something is that complicated, there should be plenty of scope for simplifying it. But the task turns out to be harder than you’d imagine.

Before he reveals some simplification strategies, he suggests people go off and try it themselves, and offers a template to work from. He has even posted a couple of examples of solutions that people came up with

Giles outlines four basic categories that all the solutions he’s seen fall into.

  • Remove – get rid of all the unnecessary controls until the device is stripped back to it’s essential functions
  • Organize – arrange the buttons into groups that make more sense
  • Hide – hide all but the most important buttons behind a hatch so that the less frequently used buttons don’t distract people
  • Displace – create a very simple remote control with a few basic features and control the rest via a menu on the TV screen, displacing the complexity of the remote control to the TV.

Some people, he says, do a little of each but everyone picks a primary strategy. Each have strengths and weaknesses, and he says that those four strategies work whether you’re looking at a something large, like an entire website, or something small, like an individual page. He goes on to describe each of those four strategies in more detail, and says that a big part of success comes from choosing the right strategy for the problem at hand.

Here is where I’ll let those people who are interested in learning more about those strategies go get the book

For people who own an Apple TV, you’ll notice they really embraced the displace strategy. Their remote is really nice, same with Roku, and other modern device makers – they have a simple device that displaces most of the functionality to the screen.

Those are nice, but there is a company that thought there might be a better way to solve this problem, and part of their app includes some IUI.  

The company is named Peel, and they built a smart remote control. They didn’t follow any of the four strategies that Giles outlined, they got rid of the remote altogether and put in on smartphones and tablets. They’re obviously not the only company to do that part, Logitech did the same thing, as did others, including some TV manufacturers and cable providers.

What makes the Peel remote so interesting is the interface is that they completely reimagined what a remote control could be. They brought the content down to the device, so it isn’t just a bunch of buttons with alphanumeric characters on it. They actually display imagery for the show, like the poster art for a movie or channel logos for networks.

Although a few years old, there was a report from Nielsen back in 2016 that indicated that the average consumer only watches about 19 total channels, or about 10% of the channels available to them. From an intelligence standpoint, it wouldn’t take long for a system to learn the ~20 channels someone watches regularly and make those the primary channels displayed in the interface.

But Peel goes even further, they add smart recommendations.

Instead of making people channel surf, they actually make recommendations of shows to watch. I’m not really sure the efficacy of those recommendations as I’m not much of a TV person (some news, some soccer, stream movies, etc.). Regardless of how good the recommendations are today, it’s hard to argue that the UX of the Peel remote is pretty great and, recommendations can always improve. Don’t get me wrong, I’m not suggesting that recommendations are easy, I’m sure Netflix has spent a ton of money on this, including their $1M dollar competition but consider what recommendations mean in the sense of a remote control…

Think about the access they have to a lot of behavioral information, what channels people tune into, how often they change channels, recurrence (same channel at some repeating interval), etc..

As a simplistic example, they know that for the last few weeks, on Monday night, that a person has changed the channel to ESPN at roughly the same time, so about 30 minute before that time, they could display some graphic for what is coming on at the time that person normally tunes in. It wouldn’t take long for a smart system to learn about the seasonality of sports, and stop suggesting it when that “program” was no longer on.

Both of those are a bare minimum of intelligence but I think still qualify for an intelligent user interface.

What do you think?