Assistant professor wins NSF CAREER award for cognitive modeling to help robots assist nurses

From robots that can assist during surgery to those that help automate laboratory processes, the idea of smart hospitals is nothing new. But what about a robot that could safely assist healthcare workers with, for example, reducing cognitive load, improving workflow, and aiding both practitioners and patients while providing safe and intuitive support?

Science and engineering are still some ways away from making this a reality. But a new National Science Foundation CAREER Award, granted to Assistant Professor Stephanie Tulk Jesso, could bring researchers one step closer. 

“When you’re considering adding artificial intelligence tools and robots and all these things that have degrees of agency, or minds of their own, there are a lot of potential ways to do that in a positive, beneficial manner,” said Tulk Jesso, a faculty member at the Thomas J. Watson College of Engineering and Applied Science’s School of Systems Science and Industrial Engineering. “Then there are a lot of opportunities to get it wrong and actually cause real harm.”

Safety and human-centered robotics design are the crux of Tulk Jesso’s new CAREER award. It is one of the NSF’s most prestigious distinctions, granted to early-career faculty poised to become leaders in research and education in their respective fields. 

At the forefront of Tulk Jesso’s $663,724 grant will be nurses, who are the critical foundation of any healthcare practice. Specifically, she aims to center nurses’ needs while designing AI platforms that will work more harmoniously and efficiently with humans.

“I am working with nurses to try and enable them to be innovators in their own environments and their own practice, and try to both get their input to design AI and robotic systems, as well as get them to a point where they’re designing their own systems,” she said. “They’re the most knowledgeable and best-positioned to create these things that would benefit their own working environment — and ultimately, society itself.”

Robots reflect humanity

Healthcare is a system with thousands of moving parts and bodies. When incorporating robots that have the potential to get in the way, make errors, or accidentally hurt others in such a field, Tulk Jesso said it’s critical that the systems are co-designed by the workers who will be using them. 

“Healthcare is an extremely complex environment in which there are so many elements of the work system: the people that are doing the job, the patients, the characteristics of the organization, all kinds of factors that will impact both the care that the patient is actually getting, as well as the working life of the individuals that put effort into that care,” she said. “It’s not like any other job. It’s a job where people are routinely seeing other humans suffering and dying. There’s a lot that becomes really socially complex, really quickly.”

At the same time, Tulk Jesso added that our perception of what robots can do is just a reflection of our own quirks and behaviors. Helping robots to better understand the humans they are emulating could, in turn, enable scientists to better comprehend the inner workings of our own cognition.

“How much can we learn about ourselves through these robotic platforms?” Tulk Jesso said. “And then on the other side, can we make them in a way that provides demonstrable benefit to people and society? Like anything, there are a lot of ways to get that wrong. Maybe there’s not even any way to get that right. That’s why we do our science to figure that out.”

As we sense and move through the world, our perceptions and behaviors aren’t merely influenced by our minds, but also by our bodies. Thus, learning to read and understand movement and body language could offer greater insight into the mental processes and cognition behind them. Taking inspiration from how humans naturally do this can also help robots behave in ways that are more intuitive for the people they’re collaborating with. 

To accomplish this, Tulk Jesso’s CAREER project will incorporate what’s called “cognitive behavioral architecture.” This will combine novel human-computer interaction approaches with a computational model consisting of the foundational cognitive architecture ACT-R, alongside more modern AI techniques, to help robots understand the world more like humans do. With roots dating back to the 1970s, ACT-R has historically been used to model and understand human behaviors such as problem solving, learning, and perception. 

“It was a deeper connection between cognitive scientists and computer scientists in the pursuit of artificial minds that were smart like humans are,” she said. “This is a very old platform that still has a lot to offer in terms of creating systems that are appropriate, responsive, and beneficial for human environments.” 

This kind of cognitive behavioral architecture may be better-suited for the fast-paced, ever-changing environment of healthcare — where the accurate translation of healthcare knowledge into intuitive interactions can enable robots to succeed at being helpers, rather than stumbling blocks.

“Traditionally, people don’t give nurses enough credit for the cognitive work that they do on the job. They think they’re just a body that’s running around and doing work. You know these things have to get done, so they’re doing it,” Tulk Jesso said. “But they’re smart, and aware of so many different factors that are going on — like the dynamics of a patient’s condition, and what they’re doing throughout the course of the day.”

By observing how nurses work, Tulk Jesso’s robotic systems can begin to make better inferences about what human collaborators are thinking and feeling. 

All this information can guide robots in their own decision-making process, resulting in assistants made for nurses, by the nurses.

“What are the kinds of things going on in that person’s mind that could then be used to inform the robot’s mental model of that individual?” Tulk Jesso said. “If I have a better understanding of my collaborator or the person I’m assisting, if I know what they’re thinking about and understand what they’re doing, I’m in a better position to be aiding them intuitively.”

Human-centered and nurse-forward

Beyond enabling nurses to have better workflows, part of Tulk Jesso’s innovation also lies in empowering those nurses to take charge of their own workforce development. In the six years that she has been collaborating with nurses, she said she often meets nurses who don’t view themselves as innovators. 

“I hear all the time, ‘Oh, I was never very good at math, so there’s no way I could have been an engineer. I can’t do what you do,’” she said.

Despite this, excelling in the nursing profession often requires quick thinking and clever, on-the-spot solutions — traits which are not only shared with, but also integral to innovation. 

“Nurses know their routines. They know their environments. They even do a lot of different workarounds on the job. They’ll fix things. They’ll have new practices and processes they’ve had to develop just to support whatever they’re doing at the time,” Tulk Jesso said. “They’re very innovative, and I firmly believe that nurses should be encouraged and trained in how to be innovators, so they feel comfortable and have the potential to transform their own space.”

Tulk Jesso plans to design a summer microcredential course to equip nurses with skills like coding and 3D printing, teaching basic innovative skills through the lens of arts and crafts. The training will take place for nursing students at Binghamton University, as well as at the SUNY Downstate Health Sciences University in Brooklyn, New York.

Ultimately, the process of validating AI systems like robots involves intensive planning and testing to ensure the technology is reliable and safe in real environments. On top of developing a project that puts nurses at the forefront, Tulk Jesso has devised a protocol that incrementally moves its way up to real-world implementation, from initial interviews and observations for identifying problem areas to design ideation.

This practice of iterative human-centered design, a foundational principle Tulk Jesso teaches in all her classes, allows her to ensure any platform she develops would have people’s best interests at heart. As AI grows more prevalent in the workforce, she said human-centered design is a process that’ll be crucial in any field where workers might be interacting with machines.

But in the meantime, Tulk Jesso hopes to play her part in establishing safer and smarter healthcare spaces. 

“Nurses need and deserve more support for the critical work that they do,” she said. “If I can provide them with robotic applications, AI applications, better human-computer interaction devices that make their workload more streamlined, usable, and satisfying, that would be a benefit to them and directly into society.”

Tulk Jesso’s project is part of the NSF’s Mind, Machine, and Motor Nexus (M3X) program, which funds research enhancing the safety and productivity of interactions between humans and intelligent engineered systems. 

“The M3X program was introduced to me by Mike Jacobson from the Office of Strategic Research Initiatives [at Binghamton University],” she said. “When I saw mind, machine, and motor nexus, that was so cool and representative of the way I envisioned the science I was interested in. I’m just really lucky that I get some guarantee that I get to do what I want to do for five years.”