Good

a&ernoon and thanks for having me here. In this talk I want to look at the
design challenges of systems that an9cipate users needs and then act on them. That
means that it sits at the intersec9on of the internet of things, user experience design
and machine learning, which to me is new territory for designers who may have dealt
with one of those disciplines before, but rarely all three at once.
The talk is divided into several parts: it starts with an overview of how I think Internet
of Things devices are primarily components of services, rather than being selfcontained experiences, how predic9ve analy9cs enables key components of those
services, and then I nish by trying to to iden9fy use experience issues around
predic9ve behavior and sugges9ons for paDerns to ameliorate those issues.

A couple of caveats:
- I focus almost exclusively on the consumer internet of things. Although predic9ve
analy9cs is an important part of the Industrial Internet of Things for things like
predic9ve maintenance, I feel its REALLY key to the consumer IoT because of what
experiences it creates for people.
- I want to point out that few if any of the issues I raise are new. Though the term
internet of things is hot right now, the ideas have been discussed in research circles
for more than 20 years. Search for ubiquitous compu9ng, ambient intelligence,
and pervasive compu9ng and itll help you keep from reinven9ng the wheel.
- Finally, most of my slides dont have words on them, so Ill make the complete deck
with a transcript available as soon Im done.

Let me begin by telling you a bit about my background. I m a user experience

designer. I was one of the first professional Web designers. This is the navigation for a
hot sauce shopping site I designed in the spring of 1994.

Ive also worked on the user experience design of a lot of consumer electronics
products from companies youve probably heard of.

I wrote a couple of books based on my experience as a designer. One is a cookbook

of user research methods, and the second describes what I think are some of the
core concerns when designing networked computa9onal devices.

I also cofounded a couple of companies. The rst, Adap9ve Path, youre familiar with,
and with the second one, ThingM, I got deep into developing hardware.

Today I work for PARC, the famous hardware, so&ware and AI research lab, as a
principal in its Innova9on Services group, which is PARCs consul9ng arm. We help
companies reduce the risk of adop9ng novel technologies using a mix of
ethnographic research, user experience design and innova9on strategy.

I want start by focusing on what I feel is a key aspect of consumer IoT thats o&en
missed when people focus on the hardware of the IoT, which is that consumer IoT
products have a very dierent business model than tradi9onal consumer electronics.
Tradi9onally, a company made an electronic product, say a turntable, they found
people to sell it for them, they adver9sed it and people bought it. That was
tradi9onally the end of the companys rela9onship with the consumer un9l that
person bought another thing, and all of the value of the rela9onship was in the
device. With the IoT, the sale of the device is just the beginning of the rela9onship
and holds almost no value for either the customer or the manufacturer. Let me
explain

As value shifts to services, the devices, software applications and websites used to
access itits avatarsbecome secondary. A camera becomes a really good appliance
for taking photos for Flickr, while a TV becomes a nice Flickr display that you dont
have to log into every time, and a phone becomes a convenient way to take your Flickr
pictures on the road.
Hardware becomes simultaneously more specialized and devalued as users see
through each device to the service it represents. The hardware exists to get better
value out of the service.

Amazon really gets this. Here s a telling older ad from Amazon for the Kindle. Its
saying Look, use whatever device you want. We don t care, as long you stay loyal to
our service. You can buy our specialized devices, but you don t have to.

When Fire was released 3 years ago, Je Bezos even called it a service.

NetFlix is another good media example. It feels natural to pause a movie on one
device and con9nue it on another because from your perspec9ve theres only ONE
Neflix. Dropbox creates this for les, Evernote for notes, and Angry Birds for score
synchroniza9on. The service is where your aDen9on is, the device is there to give you
access to the service, but other than a convenient form factor, the hardware is largely
disposable.

10

Most large-scale IoT products are service avatars. They use specialized sensors and
actuators to support a service, but have liDle valueor dont work at allwithout
the suppor9ng service. Smart Things clearly stated its service oering right up front
on their site. The rst thing they say about their product line is not what the
func9onality is, but what eect their service will achieve for their customers. Their
hardware products func9onality, how they will technically sa9sfy the service
promise, is almost an a&erthought.

11

Compare that to X10, their spiritual predecessor thats been in the business for more
than 20 years. All that X10 tells is you is what the devices are, not what the service
will accomplish for you. I dont even know if there IS a service. Why should I care that
they have modules? I shouldnt, and I dont.

12

So what do these services oer?

13

Simple connec9vity helps when youre trying to maximize the eciency of a xed
process, but thats not a problem that most people have. Weve been able to simply
connect various devices to a computer since a Tandy Color Computers could lights o
and on over X10 in 1983. That wasnt very useful then, and its not very useful now.
You can replace the Tandy with an iPhone and the lamp with a washing machine and
you get the value proposi9on of most simple connected devices. Thats not
interes9ng.

14

I think the real consumer value connected services oer is their ability to make sense
of the world on peoples behalf, to reduce peoples cogni9ve load, rather than
increasing it, by allowing them to interact with devices at a higher level than simple
telemetry and control. Fundamentally, humans are good paDern matchers at certain
things, but were not built to collect and make sense of huge amounts of data or to
ar9culate our needs as complex systems of mutually interdependent components.
Computers are great at it. They can make sta9s9cal models from many data sources
across space and 9me and then try to maximizes the probability of a desired
outcome. A person programming a device can express what theyre familiar with, or
try to create an abstrac9on based on their past experience, some9mes with
considerable skill, but a model learned from the outcome of thousands of situa9ons
across many people and long periods of 9me can compensate for much wider variety
of situa9ons in a more nuanced way than an individuals perspec9ve will ever be able
to.

15

Predic9on is at the heart of the value proposi9on many of the most compelling IoT
products oering, star9ng with the Nest. The Nest says that it knows you. How does
it know you? It predicts what youre going to want based on your past behavior.

16

Amazons Echo speaker says its con9nually learning. How is that? Predic9ve
analy9cs, predic9ve machine learning.

17

The Birdi smart smoke alarm says it will learn over 9me, which is again the same
thing.

18

Jaguar comes right out with it. They even obliquely reference the 40 years of ar9cial
intelligence research that powers predic9ve analy9cs by calling their car not just
learning, but learning AND intelligent.

19

The Edyn plant watering system adapts to every change. What is that adapta9on?
Predic9ve analy9cs.

20

Canary, a home security service.

21

Heres foobot, an air quality service.

22

Predic9ve behavior creates a preDy seduc9ve world of espresso machines that start
brewing as youre thinking its a good 9me for coee, and reorder your favorite
blend; oce lights that dim when its sunny, power is cheap and youre not doing
anything that needs them; and food truck caravans that show up just as the crowd in
the park is genng hungry. The problem is that although the value proposi9on is of a
beDer user experience, its unspecic in the details. Exactly how will our experience
of the world, our ability to use all the collected data, become more ecient and
more pleasurable?

Now Id like to oer some ini9al thoughts on user experience design for predic9ve
analy9cs for the internet of things. Were s9ll early in our understanding of design for
predic9ve devices, so right now the problems are worse than solu9ons and I want to
start by ar9cula9ng the issues Ive observed in our work.

23

Weve never had mechanical things that make signicant decisions on their own, the
rst major issue is around expecta9ons. As devices adapt their behavior, how will
they communicate that theyre doing so? Do we treat them like animals? Do we s9ck
a sign on them that says adap9ng, like the light on a video camera says
recording? Should my chair vibrate when adjus9ng to my posture? How will users,
or just passers-by, know which things adapt and which merely behave? I could end
up sinng uncomfortable for a long 9me before realizing my chair doesnt adapt on its
own. How should smart devices set the expecta9on that they may behave dierently
in what appears to people as an iden9cal set of circumstances?

Chair by Raaello D'Andrea, MaD Donovan and Max Dean.

24

The irony in predic9ve systems is that theyre preDy unpredictable, at least at rst.
When machine learning systems are new, theyre o&en inaccurate and
unpredictable, which is not what we expect from our digital devices. 60%-70%
accuracy is typical for a rst pass, but even 90% accuracy isnt enough for a predic9ve
system to feel right, since if its making decisions all the 9me, its going to be making
mistakes all the 9me, too. Its ne if your house is a couple of degrees cooler than
youd like, but what if your wheelchair refuses to go to a drinking fountain next to a
door because its been trained on doors and it cant tell thats not what you mean in
this one instance? For all the 9mes a system gets it right, its on the mistakes that we
judge it and a couple such instances can shaDer peoples condence. A liDle doubt
about whether a system is going to do the right thing is enough to turn a UX thats
right most of the 9me into one thats more trouble than its worth. When that
happens, youve more than likely lost your customer.

Sooner than we think, inaccurate predic9ve behavior isnt going to be an isolated
incident, its going to be the norm. When there are 100 connected devices
simultaneously ac9ng on predic9ons and each is 99% accurate, then one is always
wrong. So the problem is: How can you design a user experience to make a device
s9ll func9onal, s9ll valuable, s9ll fun, even when its spewing junk behavior? How can
you design for uncertainty?

25

The last issue comes as a result of the previous two: control. How can we control
these devices, when their behavior is by deni9on sta9s9cal and unpredictable by
humans?

On the one hand you can mangle your devices predic9ve behavior by giving it too
much data. When I visited Nest once they told me that none of the Nests in their
oce worked well because theyre constantly ddling with them. In machine learning
this is called overtraining. The other hand, if I have no direct way to control it other
than through my own behavior, how do I adjust it? Amazon and Neflixs
recommenda9on systems, which is a kind of predic9ve analy9cs system, give you
some context about why they recommended something, but what do I do when my
only interface is a garden hose?

26

Here are 4 paDerns Ive observed in developing predic9ve systems that I think map to
the IoT. For most of these Im going to be using examples from Nest and
recommender systems like Amazons, Googles and Neflixs which have been using
similar predic9ve technologies for years and have addressed some of these issues.

27

My rst paDern is a metapaDern. Please have a user model, a user-facing story, for
every stage of the machine learning and predic9on process, even if its a step that is
invisible to users and customers. [Acquire] How will you incen9vize people to add
data to the system at all? Why should I upload my cars dashcam video to your trac
predic9on system EVERY DAY? [Extract] How will you communicate youre extrac9ng
features? Google speech to text shows par9al phrases as youre speaking into it, and
visibly corrects itself. That UI that simultaneously tells users its pulling informa9on
out of their speech and it trains them in how to meet the algorithm halfway.
[Classify] How do machine-generated classica9ons compare to peoples
organiza9on of the same phenomena? Etcetera Etcetera

Because machine learning and predic9on are so novel and so many of the steps are
intertwined, you need to care about the UX in every single step of the process.

28

When dealing with a person or an animal, unpredictable behavior is expected and

tolerated, but today we expect digital systems will behave consistently and the
reasons for their behavior will be clear. Neither of these is true for the UX of
predic9ve systems, which dont necessarily behave iden9cally in similar
circumstances, which change their behavior over 9me, and in which the reasons for
the behavior may not be obvious.

A predic9ve UX rst needs to explain the nature of the device, to describe it is trying
to predict, that its trying to adapt, that its going to some9mes be wrong, to explain
how its learning, and how long itll take before it crosses over from crea9ng more
trouble than benet.

Google Now describes why a certain kind of content was selected, which sets the
expecta9on that the system will recommend other things based on other kinds of
content youve requested. Nests FAQ explains you shouldnt expect your thermostat
to make a model of when youre home or not un9l its been opera9ng for a week or
so.

29

Predic9ve behavior about sequences of ac9vi9es. Many predic9ve UX issues around

expecta9ons and uncertainty have 9me as their basis: what were you expec9ng to
happen and why. If it didnt happen, why? If something else happened, or it
happened at an unexpected 9me, why did that happen?

Telling that a device has acted on your behalf, and that its going to actand HOW
its going to actin the future gives people a model of how its working and reduces
uncertainty. Nest, for example, has a calendar of its expected behavior, and it shows
that its ac9ng on your behalf to change the temperature, and when you can expect
that temperature will be reached.

30

You have to give people a clear way to teach the system and tell it when its model is
wrong. Sta9s9cal systems, by deni9on, dont have simple rules that can be changed.
There arent obvious handles to turn or dials to adjust, because everything is
probabilis9c. If the model is made from data collected by several devices, which
device should I interact with to get it to change its behavior? Google Now asks
whether I want more informa9on from a site I visited, Amazon shows a explana9on
of why it gave me a sugges9on. Mapping this to the consumer IoT means way more
explana9on than were currently genng, which is either that a thing has happened,
or it hasnt.

31

Finally, dont automate. These systems shouldnt try to replace people, but to
support them, to augment and extent their capabili9es, not to replace them.

Meshre is a social media engagement tool that has a machine learning assistant
called Ember that doesnt try to replace the social media manager. Instead it
manages the managers todo list. It adds things that it thinks are going to be
interes9ng, deletes old things, and repriori9zes the managers list based on what it
thinks is important.

Ember augments the capabili9es of the social media manager. It helps that person
focus on whats important so that they can be smarter about their decisions. It
doesnt try to be smarter than they are.

32

Finally, for me the IoT is not about the things, but the experience created by the
services for which the things are avatars.

33

Machine learning algorithms used to be strictly behind-the-scenes, but in the IoT they
are actors in our lives, so as designers its our responsibility to understand the
situa9ons where the algorithms and the devices they control interact with peoples
lives, especially since theres a deep symbio9c rela9onship between the data that
comprises the models, the behavior those models induce and the people who are the
intended beneciaries.

Ul9mately we are using these tools to extend our capabili9es, to use the digital world
as an extension of our minds. To do that well we have to respect that as interes9ng
and powerful as these technologies are, they are s9ll in their infancy, and our job as
entrepreneurs, developers and designers will be to create systems, services, that help
people, rather than adding extra work in the name of simplis9c automa9on. What we
want to create is a symbio9c rela9onship where we, and our predic9ve systems,
work together to create a world that provides the most value, for the least cost, for
the most people, for the longest 9me.

34

Thank you.

35