Q&A: Slavko Milekic, University of the Arts

Amid a sea of do-it-yourselfers, Slavko Milekic was peppered with questions on Valentine's Day after a lecture he gave at a MakePhilly event at the University of the Arts in Philadelphia.

The buzz revolved around Milekic's work in developing an inexpensive, webcam-based eyetracker. It might seem futuristic--a device that allows your computer to know exactly what you're looking at and respond in kind--but it's not far off. And Milekic, a cognitive science/digital design professor at U of Arts who holds a medical degree and a PhD in experimental psychology, is convinced he and his team of students can make it happen and have a major impact on many populations.

Milekic's touch-screen based child-friendly digital environment called KiddyFace has been implemented at various museums throughout the U.S. and he has received multiple grants in the area of psychology of human/computer interaction. He'll present a paper on the inexpensive, webcam-based eyetracker at an international conference, and he talked to Keystone Edge this week about the technology's power.

Keystone Edge: What's the basis for your research?
Slavko Milekic: My expertise and passion is mostly formed by a field called ecological psychological founded by James Gibson in the 1960s. It's a branch of psychology that puts emphasis on an organism as part of an environment emphasizing that it did not evolve on its own, but in the context of its environment, thus our biological makeup is totally in accord with what we do in our environment. We're hardwired with all kinds of knowledge we take for granted because that's how we evolved. My main focus, along with my interest in digital media, is how to interface this biological role with the digital domain, but not in a traditional way like with a keyboard and a mouse. They are really inappropriate devices that served their function in the early stages of digital media. It's possible now to create environments that would actually react to humans.

KE: How did you get involved in touch-screen technology?
SM: About 17 years ago, when my children were very young, it was exactly the time that the digital revolution was happening. Also at that time, many museums started digitizing their collections and making them available. An art museum might have 200,000 works of art available, but they're not accessible to many people, including my children. This prompted me to try and develop interfaces that were touch-screen based and now even children as young as 2 years old can actually explore works of art. One of my first designs was something used in a daycare center where I digitized 50,000 works of art and created this interface that's very much like the iPhone interface, where children can browse through all these images, stretch them or interact with them, literally put a mustache on them. They were also on the floor and accessible for children so they could crawl on them.

To the surprise of educators who were anti-computer because of video games, kids actually spent a lot of time playing with these works of art, which to me makes total sense. I presented a paper on it and it created a huge reaction in the museum community. People from different museums approached me and wanted it. That's what brought me into programming, because I'm not a professional programmer but I'm self taught and started designing these applications.

KE: So you were thinking about the iPhone probably before Apple was?
SM: The iPhone, you can see the way it was adopted, so many people bought it, and so many applications are being developed. They definitely introduced an interface that had existed for years. Touch-screens were actually developed at the same time the first mouse was designed, so it was existing technology that nobody was really using. Up until the iPhone, touch-screens were used most like giant buttons that allowed you to press different areas of the screen. The beauty of the iPhone interface is these surfaces can detect gestures and motion and the surface is redefinable.

My research first started with making the display surface, which knows how you're touching it or where you're using your finger and the second step was the use of our gesture space, so the environment can pick up and react to your gestures. This was a second part of my research, investigating gesture-based interfaces, one of which is installed at the Phoenix Museum of Art, where you just wave your hand and projected images appear on the wall. If one wants to move it even further, I was thinking, it would be to design environments that can actually read your mind and know what you're thinking. That's where eye-tracking came in.

KE: How has your eye-tracking research developed?
SM: I was lucky when I was a grad student at the University of Connecticut, one of my duties was to set up their eye-tracking lab. At that time I was very unhappy because there were probably only 12 people in the U.S. who knew anything about eye-tracking. People literally had to bite a bar to keep their heads from moving so their eyes could be tracked. One of my duties was to make the bite bars, but years have passed and computers became much faster and eye-trackers became smaller.

With all this expertise, I wanted to push it to the next level, so the eye-tracking technology could literally read your mind. There's an eye-tracking application used for people who speak English as a second language. When you’re reading a document on a screen and encounter a word that you're unsure of, there's a very typical pattern of eye movements would trigger the thesaurus to open. So instead of pressing buttons or clicking, it detects increased cognitive load or confusion and opens the thesaurus. That became a challenge for me, to see how far one can push this technology. Imagine the environment, whatever device you're using, that could infer higher-order intentions, like your interest, and then automatically provide you with it.

KE: Is eye-tracking technology ready for mainstream, or is it still futuristic?
SM: Touch-screens existed for 20 years and were barely used. Suddenly they're everywhere and you can do anything on them. Just five years ago it would have seemed futuristic. I think most of it is based on marketing and the willingness of some large company to take a risk.

The paper I'm presenting in April lists a half-dozen open-source software software programs that are capable of turning any webcam into an eye-tracking device. I'm interested in using eye- and gaze-tracking to actually interact in the environment.

KE: How far along are you in realizing that?
SM: One mode of using one's eyes is when I'm looking just for observation and the other is when your gaze initiates action. A patent application that the University of the Arts sponsored last summer was dealing with an original way of using eyes and gaze to interact with the environment. To put it simply, it uses eye gestures. Say you're looking rapidly just for a fraction of a second up and down, you can select an object on a screen, or maybe looking to the left will sharply zoom in on an object.

Eye-tracking is mostly used by large companies for marketing research or product design and different areas of scientific research like psycholinguistics. But it's extremely expensive; leasing one of these eye-trackers can cost up to $40,000, you have to go through a complex setup and there's no intuitive interface. They're studying the population but not enabling the population to use their eyes as an interface device. I thought the solution would be to develop inexpensive eye-tracking technology.

If it were just a piece of software you can download on a laptop, suddenly your laptop will have this added channel it can use as an input. Imagine reading the New York Times on the web and you're looking through the headlines and there's one interesting to you and just by looking at it, the article comes up on the screen. What I did with a group of University of the Arts students was created a prototype made available for free on the web, the iWriter webcam tracker, and it's proof of concept you can have a little webcam that's $20 and make it track your eyes.

---

Joe Petrucci is managing editor of Keystone Edge. Send feedback here.

To receive Keystone Edge free every week, click here.

---