Peter Iglinski: You are now listening to the UR Quadcast, the official podcast of the University of Rochester. I’m Peter Iglinski, your host for this episode. Ehsan Hoque is an assistant professor at the University of Rochester. He studies, in his own words, “human behavior through a computational lens.” Professor Hoque received his Ph.D. from MIT in 2013, the same year he joined the faculty at Rochester. In his four short years at the University, Professor Hoque has picked up numerous honors. Among others, he was named one of MIT’s top 35 innovators under the age of 35, he received a world technology award, this year he became an inaugural member of the ACM Computing Academy—ACM being the Association of Computing Machinery—and his latest honor: Science News named him one of the Scientists to Watch in 2017. Thank you for being with us today, Professor Hoque.
Ehsan Hoque: Thanks Peter, glad to be with you.
PI: First, congratulations on your latest honor. What does an award like that mean for your research and your career?
EH: It means people, who may be outside of the field, find our ideas exciting and worth pursuing. So that’s very motivatin—that whatever we are doing is catching on, people find that interesting, and it’s something that we should continue to work on.
PI: Is it easier to get research grants, for example?
EH: It’s never easy to get research grants. It’s always difficult. Especially if you want to do something that excites you, and convincing others to pay for it, it’s always difficult. But the idea is to keep trying, and maximize the chances of being successful.
PI: Your name is increasingly being known in the field. I’m assuming that helps with people who give out the research money. They take you maybe more seriously, might help with collaborations. Is that true?
EH: Well it certainly helps with collaboration, especially within the University of Rochester. Especially, we have great medical school around here, and many people deal with emotion—but in different contexts—understand our work, get in touch with us, and that really helps to take on new problems that we hadn’t thought existed.
PI: Do you get more calls from colleagues around the country, or around the world?
EH: Well we certainly get calls from the press with a desire to cover some of the work in accessible language.
PI: Now your work involves subtle human nonverbal behavior. What does that mean?
EH: Right, so as you and I are talking, we think we might be communicating through language the most, but we don’t. We are looking at each other, we are occupying space. I’m looking at every gesture you are making, and you are doing the same thing. And that dance can raise a lot more information than the language we are saying to convey. As a computer scientist, I want to understand that dance, that coordination that individuals use to interact with each other. It could be face to face, it could be perhaps a group of individuals. And how does that change even virtually, when I talk to somebody using a video conference, where the interaction is somewhat limited, but still I am relying on the person’s nonverbal cue to decide what I will do next. And nobody knows the answer, and I think that’s what keeps us busy at work.
PI: Let’s go back to the beginning. You came here from Bangladesh. What brought you to the United States in the first place?
EH: Well growing up in a tropical country, like Bangladesh, was a lot of fun. And after I finished my high school, I could predict what the rest of my career would look like if I stayed there. And I figured it’s a time to take on a new challenge, perhaps challenge myself and go somewhere else. Maybe a new language, new culture, new people, new way of learning things, and the U.S.A. seemed like the obvious choice at the time.
PI: So what do you think you would have done, if you stayed in Bangladesh?
EH: Well I’d have a very comfortable life. My dad has a business. So I probably would have taken on the business and had a very comfortable life.
PI: In what field, do you think you would’ve wound up in, the same?
EH: Difficult to say. I think the United States has a lot of resources, take on risk—high risk, high payoff problems—which probably wouldn’t have existed in Bangladesh, so I probably would’ve gotten into business, maybe be an entrepreneur.
PI: Did you come here on your own, or did the family all come?
EH: No, I just hopped on a plane, and came here when I was 18, all on my own. It was a difficult time, especially when you’re 18, trying to come to new country. I knew English as a language, but I really wasn’t good at it in terms of speaking, so there was that. In Bangladesh, I went to school in a different medium, so I studied in Bengali. And here, everything was English. So I knew all the signs and math, but I wasn’t familiar with the terminology. So I went through a lot of struggle in my first semester trying to understand. To give an example, back home you are taught that whatever is in the book, you write it, in the exam or in the assignment. You don’t even have to cite it because it’s all in the book they assigned to you. In one of my classes in the first semester, there was art class. I had to write an essay, and I took a text from the book that was assigned, but I didn’t cite it. And they accused me of plagiarism. So there was that cultural aspect, as well, that was difficult to pick up, but I did eventually.
PI: So coming here on your own, you clearly have a bold and an independent streak, which kind of suggests how you got those honors—that you’re pretty determined.
EH: I think that getting out of your comfort zone, trying something new, always challenging yourself, will always benefit you.
PI: Now you attended Truman State University and Penn State before earning your Ph.D. at MIT. Tell us about your educational background. Did you always know where you were heading?
EH: Not at all. I never knew that I would end up in Midwest when I was in Bangladesh. I probably never even heard of the state Missouri. So I went there and went fine, and I had a good time, but then, as I talked about in the previous answer, that I struggled in my first semester, taking my first programming class. Because all the students in the class had some AP programming experience, and the instructor taught the class for them, so I had no idea what computer programming… So I knew that programming is an interesting something that will get me a job, but I didn’t know anything about it. So I struggled in my first semester, and I dropped the class, first programming class. Second semester I tried again, and I was unsuccessful, ended up dropping again. Then I realized my GPA was not so great, my confidence was really down, and I figured ‘I need to seek a change, I need to go to new place versus taking this class for the third time with the same instructor.’ That’s when I transferred to Penn State University. And then, I was really lucky in my first semester, I had an instructor who assumed that students had no idea about programming, and she went very, very slow, and that’s when I realized that I really enjoy doing this, and I started excelling. So I really go back to the instructor and said thank you for what you did to me, for going slow at the time and giving me an opportunity to pick up on these exciting skills.
PI: So at that point, was it pretty much a straight shot, you just kept going in that direction, and you wound up at MIT?
EH: Not at all, there was a lot of struggle down the road. I never even imagined going to MIT, and that’s not something I planned, I just always wanted to be focused on what I was doing. So at Penn State, the story was interesting. When I went to Penn State, they told me that you cannot go to the college of engineering due to your GPA at the previous university, and at that point I said alright, that’s fine. Then for the next three semesters I ended up taking the classes that are required to go to the engineering, and I filed a petition against it: you know what, I’ve taken all the classes that any other student would have taken to go to the engineering, I’ve taken them, I’ve excelled. Would you please change your mind? And they did. And then I went to the college of engineering, and I wanted to study computer science, because I was excited about it.
At the time, there was this rule that you have to go to a website and decide which major you want to choose, and you have to click a button, “submit.” So I clicked computer science being the first major, because I really want to study it. But the button was so small, I never clicked on it. So the college came up to me and told me that you cannot study computer science because you forgot to click that button within that time we gave you. And that was a big blow, because I come this far from Truman in order to study computer science. I struggled through it, and I excelled, and now you’re telling me I cannot study what I want? So at the time I started looking into Penn State, and I realized that there was a campus called Erie within Penn State where I could transfer and get a degree in computer engineering, instead of studying electrical engineering at the main campus at Penn State. And so one day I just drove my car and went there, to Erie, to new city. I found somebody who I knew, and I then went to his apartment, and I just started living there and that was the experience at Erie. But then, you may ask, but from then, life must have been really great? No.
Erie…I arrived in the wrong semester, that means I had to take classes without having the proper prerequisite. So I was taking classes I was not prepared for. So I was really, really struggling. In some classes, I was struggling just to pass the class with a ‘C’ so I can move onto the next level. So that was a lot of struggle, but I stuck by, I went at it. I know that I have to finish this degree. I need to be able to get to the end of it. During my fourth year, for my degree, they had to do a one-year design. So first semester, you do a design. And the second semester, you do an implementation. And it’s kind of open-ended, you can do whatever you want. That’s when I really started excelling. I thought, ‘Okay, I’m gonna build a robot, that can see people, can recognize people.’ And back in 2003, or 2004, this was considered, you know, very wild. The professor who was supervising us was in disbelief, if this guy, who’s struggled to get a C in my class, who’s dreaming so big! But I didn’t let that stop me, so we went at it. We had formed a group of five, and we worked on the robot, and the main motivation for the project was open-ended, nobody was telling me what to do. I could just do whatever I want, be creative. So then we built the robot, and then it was judged by the Best Design Award, by IEEE (Institute of Electrical and Electronics Engineers) that year, and that was a big motivation as well.
After I finished my degree in Penn State, I went to University at Memphis to get my master’s, and at the time I was studying recognizing emotion from speech. It seemed like a very exciting area in the context of learning. Let’s say I am learning with the computer, the computer understands when I am frustrated so it can mediate and allow me to be a better learner. And then we published a paper, and it was time to apply for Ph.D. So I was talking to my advisor, and was like, where should I be applying for Ph.D. And he told me, well there are many places you can apply, maybe you should also consider MIT. And I immediately told him no, because I don’t want to waste my application fee. They’re probably just going to throw my application away. And he mentioned that you should always take chances in life. So I did, and I made it.
PI: You’ve said that you failed more than you’ve succeeded. What allowed you to overcome those early challenges and missteps?
EH: Well, even today I fail more than I succeed, it’s not just then. But the idea is to keep pushing, idea is to keep believing in yourself, and realizing that, okay, if I try 10 times, I’m going to fail 9 times, but there is this one time I am going to succeed. And that’s probably good enough.
PI: It’ll be glorious.
EH: It will be glorious.
PI: What scientists or thinkers influenced you, in your career?
EH: There are many people, but I think in terms of the most impact, it was my Ph.D. advisor, Roz Picard, at MIT. So many people in the field inspired me to be a good researcher, but she inspired me to be a good human. Especially when you’re getting into research, or any other field, you often can have competition. She requested to look at that competition as a collaboration. So when you think of the rest of the world who are competing in the same space, instead of competing, just think of collaboration, and then everything changes.
Similarly, when you are arguing for your own ideas, it’s not necessary to put down others. You can still be very respectful of the views that exist, and we promote yours. So those practices are really valuable for me to push my research in a way that respects others, and also that may have more acceptability.
PI: You spent time in the Midwest, Pennsylvania, Tennessee, Boston. What brought you to Rochester?
EH: When I came here for a visit, I really liked the small environment in Rochester. I felt like this is a place where I would be noticed, I can make an impact that will propagate through the entire university. And also, the city itself, I thought the quality of life that I will have here, it will be difficult to beat anywhere else. I remember when I was negotiating my salary, I was told this is the only place where you can afford a lakeview property with this salary. And I thought, yeah, but the lake is gonna be frozen half the year.
PI: But it’s beautiful when it’s frozen.
EH: It’s beautiful. The winter is energetic, but I like it.
PI: Human emotion, that’s your focus. How did that all start?
EH: From the very beginning, I was always fascinated by human emotion. Especially being able to understand people, being able to respond to nonverbal cues, being able to understand how somebody could react to what I’m saying, is fascinating. It just opens up whole new possibilities. And being able to have more knowledge, to be able to apply the knowledge I’ve learned to real-life situations, I think makes me a better colleague, also makes me a better husband and dad at home.
PI: Your MIT thesis, we need to talk about that. You provided the first evidence that interpersonal interactions can be improved with a personal assistant. Tell us about that.
EH: There’s an interesting story around it that I enjoy telling. We were attending a conference called Asperger Association of New England, and there I was demo-ing all this technology that can recognize emotion, and I was super proud of what I was doing, and there’s one gentleman who came up to me and told me that, “Yeah you’re doing all those things, but how does that help me? I mean look at me, I have difficulty making eye contact with you. Due to my difficulty, I have difficulty getting jobs. If you could provide some kind of technology that I could practice to improve my social skills that’s in the context of interview, that could change my life.” And from that conversation I went back to the lab and thought, “Okay, what could I be doing to help this population who may need some help? In addition to solving difficult equations at work.’
And from that realization we built a 3D character, we called it “My Automated Conversation CoacH,” short for MACH, who could see you, listen to you, and then respond to you in real time, and can play the role of a conversation partner, and can give you feedback on interview skills. Now in computer science, especially in the human-computer interaction, we not just want to build a prototype, we want to validate this to ensure that whatever I’m building is serving its intended purpose. So we ran intervention with ninety students from MIT, they’re undergrad students, junior, native speakers of English, all are looking for internships. We split them into three groups. One group was control, one group just watched their own videos, because you could argue that recording yourself and watching it is good enough to improve your skills, and the third group got the intervention using MACH. And we showed that students who practiced MACH were perceived to be better candidates than the other two groups when judged by the counselors from MIT Career Services.
PI: So how does this work? Someone is practicing in the privacy of his or her own home, or office or whatever, and does the feedback come immediately, or at the very end of their practice session?
EH: That’s a great question. We have toyed with all those possibilities. We have considered the possibility of giving real-time feedback. So imagine I’m practicing as I’m talking to you, you’re giving me a lot of feedback and picking up on and deciding what to do next. That’s that. And we also talked about, okay, but that may be too much for somebody who has social difficulties to grasp, process, and then act upon. So another possibility is, you do this with the avatar, the avatar is mirroring some of the behavior. It means it’s smiling when you’re smiling, it’s nodding its head when you make a point. It’s establishing rapport by mirroring your posture. So that’s more of a real-time feedback, but in a subtle way. And then once the conversation is done, then it can show you what you did in the entire conversation immediately, so you can reflect on it. So we have toyed with all these possibilities, and there are trade-offs, design trade-offs, in terms of when you provide real-time feedback, what would that look like, because I want to help you, not paralyze you. So feedback has to be subtle. And then in the post-feedback it needs to be detailed, but not too detailed, depending on your background. So we have thought about these different possibilities have many different considerations on this space.
PI: Does the assistant groan when somebody makes a bad joke?
EH: No, we didn’t program that in. We want to make sure humans are in control.
PI: You’ve talked about your brother, who has Down syndrome. How has that affected your work?
EH: So I was blessed with a brother in 1999, who has Down syndrome. Another point, it just made me realize that, all the work that we do, if it cannot go back to something I care about, or someone I care for in my life, what’s the point? So from the very beginning I really wanted to get into applied research, where some of the stuff that we do could have real-world applications with people who may have needs. So even now, till today, whenever I take on a new project I always think about people, people like him, and the caregivers, and how technology could be useful to help these individuals and the caregivers.
PI: Has your brother used the technology?
EH: He’s nonverbal. He used some of the speech games that we built, that helps with your speaking rate and pitch and volume, but not the job interview. I think he’s far away from seeking a job at this point.
PI: Okay. How have collaborations at the University of Rochester made a difference for you?
EH: I love the fact that we are right next to a medical center. Our medical center is wonderful, the people in there are wonderful, they have access to a great set of problems, so that made a big difference in my career. Because right at this point, more than 90 percent of the projects are in collaboration with med school. For example, I give a lecture in one of the classes, and one of the undergrads used to work for Ron Epstein, who’s a leading expert in end-of-life communication. And she went and told him about me, and they invited me, I give a talk, and we’re immediately seeing potential collaboration opportunity, and we started collaborating in the space of end-of-life communication.
What it means, that, let’s say, final stage cancer patients. If you ask anyone who has final stage cancer patients, okay, where would you want to spend the last few days of your life if you were to expire, many would tell you that at home with family, being comfortable. In reality, they die being hooked up to multiple machines in hospital, because there is a communication breakdown between oncologists and patients as they deliver information. So we figured there might be a possibility to be able to use technology to understand when does the communication break down between the oncologist and the patient, and what should the doctor do in terms of delivering information in a way so that the patient is in charge of what kind of treatment they would seek that’s appropriate for the diagnosis they have.
PI: So where does that work stand?
EH: Well we just published the first paper on this, where we looked at close to 350 of this interaction between thirty oncologists and more than 360 final stage cancer patients. We looked at the conversations, especially the text data, to analyze what really matters in a conversation. And there are many exciting findings that are currently under review, but one thing that we found, was that if a doctor is brief, but overly positive, it’s not a good thing. The doctors who take the time to explain things and not very positive, for data, turned out to be rated higher.
PI: I find collaboration interesting, who gets together, how they find each other. Is it a matter of serendipity where you stumble across an individual that winds up being a collaborator, or for the most part, do you search each other out?
EH: No, it happens by chance. The idea is to meet as many people as you can, go give talks, have interesting conversations, keep following up, and you will gradually see some works out really well, and some doesn’t, and that’s alright.
PI: What will the University’s new Data Science Center mean for your research?
EH: I think data science is a wonderful initiative, to bring people under the same platform that I would probably not meet otherwise. People from the medical center, people from the Warner School of Education, people from the Eastman School of Music, people from the Simon School of Business, all these individuals who I had no way of meeting are now together under this umbrella. And that’s wonderful, because that allows us to be able to have conversations, think of ideas that intersect multiple disciplines, and how we can pursue these opportunities together. So that has been wonderful, not just for me, for everyone who are involved in this initiative.
PI: So people will learn about your work as well, and find ways to benefit from your own research and say, “Can we work together on my project?”
EH: Exactly. I can lend my expertise to, let’s say, a project in the nursing department, or maybe in the psychiatric department where they’re looking into elderly, like if you are above 75, often you lose touch with social skills, can automated assistance be helpful? Right, those connections. I made connection with Ray Dorsey about using computer version technique for diagnosis of Parkinson’s. Those connections are so interesting that I don’t think it would have happened on its own. It only happened because of Data Science Initiative.
PI: So what research projects do you have in the back of your brain, in the short-term and the long-term?
EH: I think, let’s say virtual assistant. I find that paradigm very interesting; I think they’re here to stay: Alexa, Google Home. Maybe not in its current form, but the term of virtual assistant, I think they’re here to stay. And what new applications can you enable where assistants are enabling some value in my life? Right now we have made a lot of progress in terms of recognition. I can talk to it, it’s recognizing what I’m saying, it gives me information. But can it add more value? Can I practice with virtual assistant where it gives me feedback in certain things?
Imagine the virtual assistant being the mediator of a conversation. If I’m having a group meeting, and often when you have a group meeting with people with a lot of Ph.D.’s, people disagree a lot. Can an assistant, who is impartial, who doesn’t have a personality, mediate and make sure the conversation is happening in an effective way? So there are lots of exciting opportunities with virtual assistants that we can do in the short-term. But in the long-term, as I mentioned in the beginning of this talk, I want to understand the coordination that people use when they talk to each other. There is a dance that happens. So we listen to each other, we listen to each other, gesture, we understand it, but we don’t know how it works. I want to understand how it works in my career, or maybe in the next.
PI: The final question: when you unwind, is it likely to be with or without a computer?
EH: One of my priorities with my career right now is to borrow time from my professional commitments so that I can use it to do something else, which is away from the screens. So maybe, learn how to play an instrument, or maybe do some photography, or maybe pick up a winter sport. So it’s definitely not with a computer. I want to be away from a computer, spend more time with my kids, maybe take a trip to Toronto, or maybe do some yardwork.
PI: Great. Professor Ehsan Hoque, thank you for being with me today.
EH: Thank you for inviting me.
PI: My thanks also to Kyle Tworek, our audio engineer. For the University of Rochester podcast, I’m Peter Iglinski.