As a development out of Virtual Reality, Augmented Reality and Mixed Reality have been around for a considerably long time.
A lot of experimenting has been done since Ivan Sutherland’s 1968 Head-Mounted 3D Display System and the first VR prototypes by Jaron Lanier.
Mobile phone games, like Ingress (2013) and PokemonGO (2016), were based on real-life locations utilising geo-tags. These games were played outside and inside, implementing real-life locations on earth and they added 3D elements to reality, creating virtual things in real landscapes.
As early as the 1980s, we saw several sci-fi movies that were pushing the cultural perception of Human-Computer Interfaces. These early 1980s and 90s movies were introducing bright, often blue and orange UI elements on a HUD-like screen, where the real world is perceived, analysed and enhanced with data in real-time.
Interestingly, this look of blue, green and orange lines on dark, often half-transparent displays, has not changed much in 40 years. The resolution has improved and today’s mockup displays in movies are a lot busier than they were in the 90s. But overall the look is still strongly retro-inspired, from the early days of computing in the 1970s (TRON look).
The 1995 movie “Johnny Mnemonic” explored VR elements when everyone’s idea of VR was that it would strongly distinguish itself visually from reality.
User Interfaces in “The Matrix” are not showing Augmented Reality. The entire world is a VR universe created by a computer and these old-school displays supposedly represent “reality”. These UIs were deliberately retro-styled, with green and blue fonts and lines on dark screens. This kind of perception, that real high-tech in the future must be extremely user-unfriendly, has persisted in movies ever since.
“Minority Report” used gloves as a device to interact with virtual displays, which were blending computer data with real-life views in a mixed reality environment.
The interfaces in “Oblivion” presented extremely dense data sorted in optical boxes. It’s an example of the UI where everything is visible simultaneously, thus, actually diluting information input and making it harder to navigate through decision making.
Another problem with the UIs in “Oblivion” is the practicality when screens are superimposed over real-life images. These displays are military jet fighter-HUD inspired, but they also show the problems of superimposing UI elements on real-life views with a changing background and brightness iterations.
The User Interfaces of “Iron Man” were mostly completely out of touch and didn’t seem to make any sense or serving any function at all. Which was perhaps the point to highlight Tony Stark’s genius with mystery. Again, we see predominantly blue and orange lines on dark background patterns on HUD interfaces.
“TRON Legacy” made no exception to the blue- and orange look. Taking a cue from its 1980s predecessor, the entire movie is based on glowing blue and orange lines.
“The Blade Runner”, 1980, was one of the first films to introduce the iconic “glowing lines on dark-background” look.
“Blade Runner 2049” introduced a patina and an even stronger 1970s retro look, reminiscent of the first Atari arcade games. But these screens are even more obscure, more blurry and made it even harder to identify information or purpose of any given element.
Of course, these UI screens look somewhat attractive and impress by making processes look highly sophisticated and complex. Part of the fascination with them is that we don’t really get what is going on.
But in real life, such complexity would be ultimately overwhelming and throwing all known data at the user wouldn’t serve a good purpose in real-life circumstances – independently of which period it is.
Billions have been invested in research by the military worldwide. Pilots have been flying fighter jets with HUD helmets for more than 20 years. But in our culture and public perception, we are still in a stage where not many people understand the differentiation between virtual, augmented, mixed or blended reality.
More than most changes affecting our interaction with computing devices, Augmented Reality is a transition from the flat, screen-based interaction towards an emphasis of the flow state, driven by a steady exchange between the user, their surrounding environment and the device between them.
There is this challenge ahead of how we will develop the language of User Interfaces in Augmented Reality environments. Pop-cultural influences aside, there are true needs in learning and a better understanding of what happens in real-time perception, analysis and decision making. These are hard UX questions that need a lot more investment in research and development.
And yet, relatively little is known about the User Experience of Augmented Reality. The Nielsen Norman Group summarises in 2016: “AR technologies can impact the user experience by decreasing interaction cost, cognitive load, and attention switching.”
The most intriguing part of Augmented Reality user interfaces may be the intersection of what happens between the user’s perception and the device augmenting their reality. This is perhaps also the most challenging part for the development of modern-day, functional displays.
It is more about a mindset and fulfilled expectations in a flow state, rather than the User Interface and what happens on the screen itself. How the augmentation of reality works with users, how it can assist the user in fulfilling their goals and meeting their expectations is the main question UX in Augmented Reality should evolve.
The UI of a “reality augmenting” device is not just a crutch. It becomes the mediator of time, space, expectations and events.
The true interface between reality, device and user lies in the user’s mind. As developers of UX for the augmented reality, it is our job to look at the user’s perception, cognitive load, assumptions, learnings and expectations. It is our job to meet the user halfway, or all the way, to converse with them, to anticipate what the user wants.
In his book “How Real is Real?” (1977), professor Paul Watzlawick, a philosopher and psychologist, describes reality as something that is only rooted in our perception. We have no proof outside our perception that reality exists, he argues, we can only rely on what we perceive and that may not be an accurate representation of reality in itself.
The human brain is built for decoding perception. From hunting and gathering to evaluating our surrounding and assessing danger, we have an evolutionary history of making sense of our surrounding. Our minds are constantly using the previously learned to further decode and interpret the world around us.
Unfortunately, because our brains and minds prefer the linear narrative, we are also prone to deception – when something is making a thing appear different than it is, which may undermine our ability in sound decision-making. From optical illusions to the cognitive overload of information, we are very vulnerable to interpret something in a way that may have catastrophic results.
The line between enhancing and enriching our perception of reality and diluting it or overloading it with superfluous information is blurry, but it is very important to consider when creating an AR based HUD interface.
Driving a car and interpreting locations, distances, events falsely or too late may lead to accidents. Our understanding of reality is heavily influenced by bias and the cognitive load – how much of “reality” we can process at a given time.
Ideally, HUDs and Augmented Reality could support us by providing better guidelines in real-time to assist us with time-critical decision making.
The perception of our surroundings changes steadily. Especially in a car, as a driver, we are making new decisions based on exterior input every moment while we’re driving. We see street signs, other cars and objects, buildings and landmarks, and everything is registered as input to make instant decisions.
A device employing AR should help us to respond contextually to new external information and account for changes to our environments. We should be able to use it with minimal to no input while driving. And it shouldn’t get in the way or restrict us from operating the vehicle.
Screen graphics in movies have injected a retro-inspired style into pop culture. However, in regards to usability, these User Interfaces are utterly useless. They are mostly obscure and the displayed information is not identifiable. Most obviously, virtually all information is present at all times – which is opposing the advantages of an adaptive, conversational and anticipatory UI, which is in steady flow and exchange with the user.
Augmented Reality interfaces today represent more closely “non-command user interfaces”. Tasks are accomplished with the help of contextual information collected by the computer system, not only through commands explicitly provided by the user.
Our immediate surrounding is full of tactile interaction points: a door grip, a seat adjustment, a steering wheel, the ignition button, the rear mirror – these are just a fraction of real-life cues and interactive points we use to navigate our way throughout a user journey, trying to fulfil our intent.
When it comes to interaction with user interfaces, humans are more capable to adapt than we might assume. The developments of the past couple of years have shown how generations of users grow up with different environments, from Windows to macOS and iOS to Android, and they quickly pick up the variances of these different operating systems.
Humans have the ability to comprehend through learnings, we can program our brains with learned experiences. We are not only basing our learnings on what we already know but transcending to the unknown and making conclusions, solving issues we have never encountered before.
This is why video games are easy to learn, even though they all follow different rules and have not such a unified surface like the expected patterns of an operating system. Each game makes using the UI a game in itself. And that is the same driver of what happened with other interfaces as well.
We can’t make things too hard for users. They need a chance to learn and adapt fluidly, as they are going. Micro-rewards in form of confirmation and showing the user is getting somewhere is building trust, it’s making the user feel they are on the right path and making progress.
Particularly HUDs with user interface elements aiding the driver by adding information to the driver’s view, it is difficult to keep the look and feel balanced, as light may shine through the window, or light might interfere with the bright UI elements on the HUD. Basically no chosen colour can counter the impact of direct light shining through the shield window into the vehicle.
With a user interface in a practical AR application the challenge lies in making the UI blend into the environment, at the same time to pick up on real-life cues and enhance them, as well as stand out when it requires the user’s attention.
This is a
Intent is born out of an expectation: A person expects to get training shoes in a sports clothing or shoe store, so going there with the intent to get a pair of trainers is a direct result of the person’s set of expectations. But beyond “getting new shoes”, the expectation set may be more varied and complex: a certain, but vaguely defined style, a certain array of colour preferences, an overall look inspired by previously owned models, what friends or spouses are wearing, etc.
Expectations are always a precondition for intentions. Expectations are informed by what we already know, the experiences we’ve made – but also by advertising, by user tutorials, how the store presents itself, its reputation and the experiences other people might have had – hear-say, comments on social media platforms, video- and blog reviews…
In short, expectations are in a steady flow and a simplified result of complex input from various sources. User expectations are, by nature, strongly directed by bias. To know what we want, we first need to form an opinion and this leads to bias, the idea that something must be as we expect it to be, with little room for differences.
But we are still able to accept and even anticipate differences to our expectations based on what we’ve learned because we have learned a language of what makes our expectations work.
The same principle applies to user interfaces: If my expectations are fulfilled and the UI guides and assists me in achieving my intent, over time my trust in this application will grow and I will rely on it to fulfil my expectations.
Google Glass was the first HUD User Interface based device posed to hit the mass market. Due to the small market adoption and a disastrous marketing campaign, the product was removed from the market, quickly after it had been introduced.
The main problem was, people couldn’t imagine how and why to use these things in public. Google Glass didn’t seem to tightly integrate into life and given the low capabilities of AI and machine learning at the time, no substantial use case was convincing enough to persuade an audience to invest in this gadget.
As it nearly always is the case with innovative products, it did attract an audience of geeks and first-adopters. Feedback from this testing audience proves to Google that the product was not ready yet for the market and people had not been waiting for glasses they are wearing all the time, every day.
While driving, users looking at a HUD display have to decode both, the depiction of reality in front of them (the street and elements surrounding or moving on the street), as well as the information added to the view (information, data points, questions, decision points, assisted wayfinding and guidance).
So adding information in a steady flow by augmenting reality means an increase of cognitive load. It is crucial to balance out the flow of what can be processed in real-time interaction with reality and the HUD user interface.
If we are moving forward while looking through a HUD user interface, we are experiencing events on a timeline. Our brain is already tuned to accept and process incoming information in rapid flow. It is constantly challenged to output reasonable perception and decision making.
Throughout time (the timeline as we are moving forward) and space (the space in which we move), our interaction is never static and screen-based only. Most of the processing happens internally in our mindset.
A conversational UI asks the user questions, based on intelligent assumptions or guesses about what the user might want to know or do next. Chatbots have been trending in customer support, but they are just the tip of the iceberg of actually machine-learning-driven conversational user interfaces.
A conversational user interface frees the user of thinking about correct input or further processing. It allows users to act largely naturally and humane, and not have to adapt their input to match a given format.
Part of this is enabled by adapting to the user’s needs and intentions. The virtual assistants Alexa and Google Voice cannot only understand initial questions but draw context out of further questions within a limited set of variables, based on machine learning and assumptions about contextual relevance.
Another example of adaptive user interfaces are feeds, such as those applied by Netflix, Spotify, Instagram and Facebook. These feeds change constantly with every interaction you make. They don’t converse with you directly, but they have machine learning algorithms driving what comes up next on your feed, based on several variables you consciously or subconsciously influence. For example, if you are watching a video, depending on how long you are staying to watch, this will influence further items presented to you in the stream.
An adaptive interface is the first step, it changes according to your actions and decisions. One step further goes the anticipatory user interface: it cannot only adapt but more accurately foresee what your next interaction or decision might be like.
Anticipatory user interfaces are the holy grail of the upcoming revolution in AR interfaces. Users will experience interfaces which seem to have an almost magical, empathic quality to them because they will be able to present information not only upon request but even before the user thought of it.
Just one typical user experience with Facebook on a desktop browser:
You can see how user intent is not the driver for the experience here.
Next time you think about how we got to this place, with people having the attention span of a fruit fly, remember how it was made this way by design.
Remember that Facebook is not just “a tech company”, as you often hear it referred to. Facebook is foremost an advertising company. Its whole business model is about selling ad space to companies, so they can book ads on Facebook that are specifically targeted towards you. To better tailor these ads to your interests, Facebook needs as much data as it can get from you – inside Facebook.
The ads are on Facebook too, so Facebook has no interest to have you share anything outside of Facebook.
There are ways around Facebook’s anti-user design. But these involve steps that are inconvenient – something Facebook is betting you don’t want to do. For instance, you can go to a search engine (preferably one that doesn’t track you for advertising) and look up that video, or other items you wanted to share.
Or, if you want to actually share the Facebook post but outside Facebook, you could look up the page’s code (using the page inspector) and find the link to the actual post.
Or you use the mobile Twitter app and choose “open in Safari”, or “copy link”. In any case, on the desktop experience you’re locked in more or less unless you become elaborate to work around the fence.
Let’s say you made an awesome video about your product. Now you want the world to see it and you want a lot of people to come to your website, check out your product and of course, you want people to buy your product.
But how do you generate an audience? The goal is to get as many eyeballs as possible, right? Or is it? Wouldn’t it be great if interest in your video picks up and you wake up the
It’s either happening or it’s not. And no “growth marketing hack” can change this. If you rely on growth-hacking methods, you will get more viewers, but less of the audience you’re actually interested in. Every new viewer will come with a different expectation. They won’t see your video because they are really interested in getting your product.
What happens when a product goes viral is that it has something people want to share. It is well done, has a really touching, or a funny, or a moving and a compelling story, something we want to share with our peers.
The best (and only) way you can influence the viral effect with a sustainable audience is by actually making your video touching, or funny, moving, providing a compelling story.
Of course, sensational or slapstick clips that have nothing to do with your product can grab attention too, and if it is click-bait, you can be sure it will be shared. But how will this help your product and your sales? More audience of any kind does not translate into more sales.
You need an audience that is interested in what you have to offer. This audience is open to listen to what you have to say.
Fancy animation graphics, or a cool guitar song, or a peppy presenter are only tools – it’s not what will make people want to share the video. The secret lies in how you dramatise your story. How well you bring out the core that makes your product special. The essence of how people experience it.
Provide a story and a message the audience can identify with, then your video has the highest chance to be shared.
Clicking or tapping the points will check them off from the list:
Do you want to learn more? Let us know your questions in the comments and we’ll do our best to answer them.
“Who do we want our clients to be?” my question turned the room silent. Besides me, there were five people. The CEO, the managing director, the two unit leads and the strategy director. “What do you mean, the business size?” the CEO asked me.
I was talking about the type of clients that are right for us, as an agency. It was in the early days of “digital”, the term that made advertising agencies cringe and drool at the same time. I just had been hired as a “digital director”, a role, I was assured by the CEO, that would entail strategy and business decisions. Shortly after my entry, they added me to the management board of the company.
The clients we want, I explained my thinking, are the ones we are proud of. Not because they make us rich, but because they are growing with us. We want
The clients we want, I continued, are the ones who understand that it’s about the experience people have, not forcing them to buy things they don’t need.
Had I won over half of the team with the first part of this explanation, I lost all ground with the last part. Experiences? What the hell was he talking about?
“Sure.” the CEO nodded and then looked away from me. “That’s a given. Of course we want the clients of our clients to like the products of our clients. But that’s not in our power.”
“Then why am I here?” – That’s not what I said, but what I thought. It felt futile to engage in further discussion.
They were ad people. All
“Digital” was seen as yet another channel to pump out messages over Flash banners and “engaging pop-ups”. Who doesn’t like those ads blocking the way to get what you came for? The ad business was successful. It made multi-billion dollar contracts based on latest edge CGI in TV commercials and colourful print ads. It wasn’t going to be in trouble any time soon.
Or was it?
This happened almost 10 years ago. In Internet years, that’s about a hundred years. Yet the story has relevance. It has taken place hundreds of times, in hundreds of marketing businesses and ad agencies. Some of these companies are still around, some have learned to adapt, or they simply bought enough “digital marketers” to allow them to change shape, with the entire industry changing shape alongside with them.
But roughly 50% of the companies I knew at the time 10 years ago have been merged with other networks, sold off or they simply seized to exist. And the problem wasn’t that they were unsuccessful, or that they didn’t hire people “who get digital”. The actual problem was that they were blind for the big picture.
Digital – the Internet – is not just “another channel” you can use to pump out advertisement. And while a lot of businesses still rely on advertising as their main business model – quite successfully so – just about everyone in the “digital world”, startups, tech companies, even ad agencies – they are all aware now that they need to find different ways to make money. Enter the subscription model: Really good stuff that’s worth something.
Of course, promoting messages and marketing products and services is something that won’t go away for as long there is a capitalist economy and a society buying services and products. But the days of “creating magic” with funny, loud ads, and spending disproportional amounts of money to be seen and heard, those days are numbered.
The clients we want are striving for change and improvement. They are looking at the big picture. They understand that society is in a process of change and adaptation to new paradigms, and no business or industry is meant to stay the same forever. They also understand that exploitative models, or schemes an industry is based on, the idea of fooling people to believe that every business has sincere intentions and authentic products, are not really sustainable models.
See, a business, when you make money out of something, is sustainable when you are not cheating on people. If your business is built on making a quick buck but not caring about what people need, or who they are, what they are living for – what they want from life – your business is ultimately short-lived.
There are numerous examples, especially from the last thirty years. After the 1980s, the decade that cracked the secret of exploiting people through their hopes and dreams, everything sobered down. And after the last property business and stock exchange crisis, everyone knows that chasing happiness in a capitalist-driven society is basically running after unicorns. (Is it a wonder, the very few successful multi-billion dollar companies are called ‘unicorns’ among venture capitalists?)
So we are not that kind of company. We are also not overly romantic, or idealistic, and we don’t believe in fairy tales. That doesn’t mean data is the only source of truth for us, but it’s a pretty good indicator for where the wind is blowing.
To answer whom we want as clients, we had to answer who we want to be. That’s not as easy as deciding what you don’t want to do. It means you need to confront your beliefs with realistic expectation sets, regarding feasibility, regarding business capacity and regarding growth. Especially in an economy where the market is still pretty much driven by outdated ideas about unicorns and magical growth.
I want to talk to you, if you are reading this and you’re not wondering what I’m talking about. I want to talk to you, if you are curious about what we can do with your product or service, or if you believe you guys haven’t reached the full potential for it yet. I want to talk to you, if you are looking for someone in this business who doesn’t look at things through a product engineer’s keyhole. Or that of a marketer, or a UX designer, or a branding company.
Like I said earlier: we do all that stuff. But we don’t do it thinking they are proven recipes to bring success. We also don’t buy into doing things like everyone else, just because, hey, everyone’s doing them, so they must be right.
We don’t do stuff – anything really – without reason. Without thinking about why and how we should do them. Without coming up with a strategy that fits, because it can expand, grow and change direction while we’re working with it. We don’t do stuff we don’t believe in, and we don’t want clients who don’t believe in that stuff either.
We want partners as clients, to think, to look and find out, to discuss, to search and find, and to come up with solutions they didn’t know they needed. Because unless we do all that, searching, discussing, learning and finding out what’s real, we aren’t doing a good job.
We avoid confirmation bias. Big time. We avoid wishful thinking and copying successful models, just because they were seemingly successful. We avoid falling for ideas we can’t believe in, because we did our homework and found out there is no market (also known as “zero need by people”) for them.
What we can do is working with you to build on the great ideas, products and services you have, taking them and using everything we know to design them better, to bring out their core and to design how they are experienced. But for that to work you need to drop the magical shit and start doing things right.
Credit Suisse Switzerland recently introduced a new debit card—the first of its kind by the Credit Suisse. Practically everywhere else on the planet, debit cards are very common in 2017.
The fact I can now use a debit card instead of a credit card presents a huge logistical advantage for me. I use it for online payments at Amazon, or server hosting and domain fees. It is perfect for booking flights and Airbnb stays.
I can put money on the card before I travel, use it for payments outside the country, and I always have a good overview of what is left. I can’t overdraw it, and I don’t have to
The card is actually issued by a partner of Credit Suisse, “SwissBankers”. This company made a mobile app that allows you quickly glance at your balance. It’s not a superior experience, but it does a fairly good job, and booking flights very few things are getting in the way. There are only minor issues throughout the app user experience.
On Credit Suisse’s side, things look even direr. Here are the ways you can charge the Credit Suisse debit card:
You actually have to fill out a full statement, as if it was a money transfer to a different bank, including account numbers, long confirmation numbers, address field, etc. And then, still hard to believe, it takes three full working days to confirm your online charging of the card.
How do you learn about the ways to charge your Credit Suisse debit card? You have to call them, then they send you a printed document by snail mail (three days waiting), which describes the complex and anti-user-friendly ways to put money on your card.
There is no Web page that explains this. Nothing online, but a few marketing lines, praising the advantages of the Credit Suisse debit card.
In contrast to the above, the expected user experience is quite straight forward:
This should be the same experience mobile as well as on the Web.
★☆☆☆☆
If Credit Suisse Switzerland commissioned a Service Design process, they would quickly discover the weakness of their product release. Even a simple analysis of the existing user journey would have revealed these flaws, which are breaking the customer experience.
Credit Suisse Switzerland may have technical reasons standing in the way of a straight forward, better solution. But those issues should never block the basic functions to meet expectations set by customers. A simple way to put money on their debit card is a fundamental, minimal expectation.
Releasing such a half-baked product in 2017, particularly by a well established financial organisation such as the Credit Suisse, is sabotaging the product and hurting customer trust in the brand.
The best things in life are those you are coming back to. The things you like, because they just work, or because they evoke this special feeling you enjoy. Those are the things you got fond of because when you use them more often, you find that they are always reliably good.
We make such choices with brands every day when we buy something. We decide to try new things all the time, but the things that stick with us are the things we really start trusting.
Building a product or service means you are not looking at one thing that does something, or a collection of functions. What products or services do is fulfilling the most basic expectation: A running app that measures my runs. A car device that shows me when I’m driving more economically.
The things you keep using, again and again, have more than just features. They go an extra length in delighting you, in delivering great experiences. It’s a combination of great execution and a design that puts a lot of thought into the product or service, so it seems to be able to predict what you want. The best products provide a feeling of fitting seamlessly into our lives.
This isn’t by accident. It’s a result of learning all about customers and designing the best possible experience. An experience that doesn’t just work how it was intended. It does a little more than that. It’s an experience that delights, again and again.
When you are creating a product or a service, you are probably not thinking about relationships. Not at first, at least. You are probably considering your ideas, working on your vision and – at best – you are thinking about what people need.
But the story doesn’t end with delivering what people need. If things we love would just deliver what we need, we wouldn’t end up loving them.
Of course, functionality is important. But it is not the only factor that drives our decision patterns. We live in a world that is in a steady flow, in conjunction with everything surrounding us, and we are bombarded with new product offerings every day.
On our journey through life, we want things that support our intentions. We don’t want things that get in the way. We want them to be intelligently designed, to match our expectations, or better—even exceed our expectations. Most importantly, we want to trust these things, because we are letting them in into our lives.
Our expectations in products and services are just the same as what we would expect from everyone else around us: to be nice, fair, helpful. To be with me, not against me. To help me trust them, as they are trusting me. If we meet these qualities in people, we tend to trust them and there is room for a relationship, even friendship is possible.
As a company that makes a product or offers a service, you can make that difference. You can design the experience. Everything we encounter in life is an experience, and experiences can be designed.
This is what we are all about. We want to improve what you are making, so that the people who pick up your product, people who are interested in using your service, won’t be disappointed. In fact, we want those people to be elated that they found what they always wanted.
And we want them to feel that with your service or product, when they are entering this relationship, they can rely on having great experiences, because they know you care about that too.