A professor in the department of electrical and computer engineering at the University of Toronto and an IEEE Senior Member, Steve Mann built the first wearable computerized-vision system before he started high school. Since then, he has constructed dozens of these systems and amassed more hours inventing, designing, building and wearing them than anyone else. Steve is widely regarded as “The Father of Wearable Computing.” He is also the co-author of All Reality: Virtual, Augmented, Mixed (X), Mediated (X,Y), and Multimediated Reality published by Cornell University.
How do you distinguish among augmented, mixed and virtual reality and frame conversation about this space you have helped pioneer?
Steve Mann: “Mediated reality” is the right framework because anytime we use some of these kinds of devices, regardless of whether intentionally or unintentionally, it’s an intermediary between us and the outside world. It modifies what we see. You have augmented reality, mixed reality, extended reality, expanded reality and virtual reality—they’re all special cases of a broader concept of mediated reality. There are different axes in a multidimensional space that the term “mediated reality” most accurately accounts for. (See Figure 1.)
Please tell us about some of your earliest innovations in this field.
Steve Mann: In my childhood in the 1960s and 1970s, I invented a way of seeing radio and sound waves in perfect alignment with reality. It is a lock-in amplifier designed specifically for augmented reality. I founded a company in China collaborating with a university there to manufacture this product. (See Figure 2.)
Another example is HDR—“High Dynamic Range” imaging—which I invented to help see electronic arc welding. My grandfather taught me how to weld as a child. When welding, you see only a little speck of light, and everything else is almost completely black. I thought there must be a better way of doing this, so I invented HDR to help people see. Now it’s used in many different products like Apple iPhone and the Samsung Galaxy phone. Most commercial cameras and smartphones use it, but only in the sense of taking a single picture. My original creation was for a visual stream of reality. HDR is a form of mediated reality that both augments and diminishes. It augments the dark areas of the scene where you couldn’t see anything otherwise, and it diminishes the bright intensity of the arc, giving you a mediated reality. Deliberately mediated reality diminishes certain areas of the scene and augments others. (See Figure 3.)
I’ve also created an ad filter which filters out advertising in your eyeglasses. It bleeps out ads by filtering them and changing them to useful subject matter. This form of diminished reality is good for people who are visually impaired and have trouble with confusion. It can filter down the scene into a simpler form that is easier to understand.
How do ethical considerations come into play for you with regard to mediated-reality innovation?
Steve Mann: We must look at what effect technology has on people. Technology is an intermediary between us and the real world. In what ways does it help us, and in what ways does it harm us? In this way, mediated reality is a field of study that asks questions on how this technology impacts us and how we interoperate with it.
We have a certain responsibility when designing these technologies. I think too many people come in and say, “Hey, our product is great. It’s going to make the world better.” But it doesn’t always make the world better, and, in this way, I think we need to recognize—we should recognize, we must recognize—that we should look at what effect the technology does actually have on people. Many of these products are actually modifying a person’s perception of reality, and there is the possibility that it might permanently modify certain neural pathways.
Could you give us an example of this modifying perception of reality?
Steve Mann: I built my first computerized-vision system in the 1970s, before I started high school. I’ve been wearing these types of systems for many years now. Most people are using them for only short periods of time. I’ve been wearing them constantly, every day, all day. The long-term adaptation of that creates interesting problems because you adapt to it and start to see the world that way, even when you take the eyeglass off.
In my book, Intelligent Image Processing, I define something called the “aremac”—the word comes from “camera” spelled backwards. It provides a way of presenting information to the human visual system in perfect depth alignment with reality. I found the trick to overcoming issues of eyestrain is to arrange things so that the eye can focus at any distance and still see the display clearly. This arrangement of optical components is what the aremac provides. It is basically a TV screen that you can focus at any eye distance. Without it, your eyes would accommodate to a certain focal distance, which is often at odds with the distance things actually are in the scene. For example, when you look at objects that are infinitely far away, your eyes focus to infinity.
This is why I say we must especially consider what the technology is doing for humanity. We have to see mediated reality as an intermediary, because it’s not just something that mixes into our life. These technologies are actually mediating our interactions with the world, and the concept of mediated reality encompasses smart clothes, cars, buildings and cities, because clothing is simply a building built for a single occupant. (See Figure 4.)
What is the value of organizations like IEEE in this regard?
Steve Mann: One of reasons that IEEE is perhaps one of the best organizations in the world is that it is very focused on its core mission. Like IEEE, I am dedicated to advancing technology for humanity, and, in this sense, I’ve been inspired by IEEE as a guide on how to focus my life and how to direct my research.
I think the real value of IEEE is this bigger picture—this ability to understand more holistically what all this technology is about and why it’s come into play. IEEE is the greatest technical society in the world in the way that its mission is focused on the stuff that really matters in our society. We must acknowledge that mediated reality has effects on the human user—both good and bad, both intentional and unintentional. We have to look at the human condition and the humanity of the situation.
For more information on these emerging technology areas, please watch “Mixed Reality—The Future of Our World,” see The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems’ “Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems” and get involved with IEEE Future Directions and the IEEE Standards Association (IEEE SA). Also, please visit the IEEE SA at Augmented World Expo (AWE) 2018, 30 May and 1 June 2018 at the Santa Clara (California) Convention Center.
