Jiebo Luo

Looking at the trends of pictures of what products people post and comparing these with past sales data has allowed the researchers to come up with predictions of how various products are selling.


Each Image Tells a Thousand Words

Jiebo Luo

After being a researcher at Kodak for more than 15 years, it’s not surprising that Jiebo Luo would focus his University research on images.

Now an associate professor in the computer science department, he has, like many of his colleagues, become intrigued by the challenges and possibilities that Big Data pose. But unlike many of his colleagues who are focusing on the information contained in the words and sentences on the Internet, he analyzes images.

“It really is the case that each image contains a thousand words,” said Luo to kick off his presentation at the Big Data Forum in October 2012 at Rochester. For example, the iconic painting of the Mona Lisa would “contain” visual words (i.e., visual patterns and features) like “hair,” “face,” “color,” “hands,” and maybe even “smile.”

Working with images brings its own problems, however. Not only do you have to mine the web to find images, often in diverse formats, from which you can extract information—you also need the computer to be able to understand, somehow, what the image is about.

Training a computer to identify certain features is the first step. But these features need to be unique or identifiable in some way. “When you look through the viewfinder of your digital camera as you’re about to take a picture these days, you’ll probably see little squares around everyone’s faces,” Luo says. “This is the kind of technology we use.”

He and his colleagues also try to teach a computer to identify other objects, such as a car or a castle. But it’s not easy—many of us have seen our camera insist that the tree next to the friend we are photographing is also a person. “It is easy for a human to identify a chair; we all know what they look like,” Luo adds. “But it’s nearly impossible for a computer to do so; chairs can look just too different.” So it is even more impressive that Luo and his team have been using images to predict election results.

The use of social media during the 2012 presidential campaign was widely discussed. Every new debate involved the “most ever” simultaneous tweeting—until the next one. And a lot of these media included pictures.

As the outcome of election night became clear, President Obama tweeted a picture of himself with his wife, Michelle, entitled “Four more years.” . . . That tweet went viral globally and almost instantly; within just a couple of hours it became the most retweeted message since Twitter began in March 2006. It captured the moment, and in retweeting this picture people expressed a view.

Luo explains that, in a sense, every time an image is uploaded or downloaded from the Internet it is like a vote in favor of or against a candidate or position. He finds that drawing on this information—across millions of Internet users—is like tapping into the wisdom of the crowd.

So Luo and fellow researchers taught a computer to recognize pictures of Barack Obama, Mitt Romney, and the vice presidential candidates, as well as pictures of campaign signage, and to collect these images so the researchers could analyze them. And by gathering these pictures they also get other data. When a picture is uploaded, other information is usually included, such as time and location. Putting all this information from the images together with real-world polling data as the training data and using state-of-the-art data mining techniques, they called the winner correctly in all the swing states.

But their model was more sophisticated than just whether an uploaded picture was of Obama or Romney. Accounting for things like whether the picture was meant as a criticism or endorsement of whom it represented, predictions are made more accurate.

And they have found their model can be used not only for elections but also for predictions of market sales of products.

“There might be a lot of people searching for the ‘iPhone 5’ on Google or tweeting how they really want it, but talk is cheap,” says Luo. “If you’re uploading an image of a specific product, it is quite likely you actually own it.”

Looking at the trends of pictures of what products people post and comparing these with past sales data has allowed the researchers to come up with predictions of how various products are selling.