Immersive-media research earns computer scientist an NSF CAREER Award

As humans become more integrated with their technology, augmented reality and virtual reality will become more and more a part of our daily lives. Google Glass and VR headsets are just the start.

Immersive media will require more bandwidth and storage than the video streams we watch today on services such as Netflix and YouTube. Keeping the images smooth and as lifelike as possible under limited network and storage capacities are key to immersion and usability of these emerging technologies.

Yao Liu has been working on this problem, though, and now the assistant professor in the Thomas J. Watson School of Engineering and Applied Science’s Department of Computer Science has won a five-year, $486,169 National Science Foundation CAREER Award to fund her research. The CAREER Award supports early-career faculty who have the potential to serve as academic role models in research and education.

Liu’s work centers on area-of-focus projections, which are generated by analyzing human behavior to decide which parts of a 360-degree image get most of a user’s attention. When creating the area-of-focus representation, a larger proportion of information from these important areas is funneled into the encoded version of the image.

If you are facing one way in a videogame and looking at, say, a giant golden dragon, you’re unlikely to turn around 180 degrees to see what’s behind you. You’ll want to keep your eyes on the shiny creature in front of you.

“Sometimes it’s not only predicting based on past user behavior, but also is content dependent,” said Liu, who joined the Binghamton faculty in 2013. “If we’re watching a basketball game, we most likely are looking at what’s happening on the court rather than what’s going on in the audience.”

Liu initially worked on traditional streaming video technology like we see on YouTube, but four years ago she became immersed (no pun intended) in the world of 360-degree videos where users can control where they’re looking.

She saw the research as a challenge: “When you view 360-degree videos on a head-mounted display, the video quality is pretty bad — but when I’m watching ‘traditional’ video, it is crystal-clear.”

Liu sees her work applying not just to entertainment such as films, television shows, sporting events and live music, but also suggests some educational possibilities.

“We can teleport students to a remote or virtual environment,” she said. “Maybe there are topics such as environmental protection that would benefit from this kind of immersive technology.”

Winning a CAREER Award “feels great,” she added. “This is a five-year award, so we can have a more ambitious research agenda. There are going to be many advances in networking technology and computational power that will affect our devices in the next five years, so I wanted to propose something futuristic.”

She also is grateful for the support she received from the CS department, the Watson School and Binghamton University since her arrival nearly seven years ago.

“When I joined the department, I was a fresh PhD graduate. I had an idea how to do research, but there are a lot of other aspects to being an assistant professor,” she said. “The Watson School and the University also offered a lot of help by mentoring junior faculty with grant proposal preparation and submission. This award would not have been possible without all of these colleagues around us supporting us.”

Liu’s NSF CAREER Award is “System Research to Enable Practical Immersive Streaming: From 360-Degree Towards Volumetric Video Delivery” (#1943250).