Future Benefits of AR/VR for First Responders: Five Questions for Dr. Andrew Phelps

Credit: iStock

Dr. Andrew Phelps is director of the Game Center at American University  and a professor at the Human Interface Technology Laboratory NZ at the University of Canterbury.  A highly respected researcher who works at the cutting edge of virtual and augmented reality technology, Dr. Phelps will deliver the keynote presentation at the Equipped leadership summit on March 14, 2023. His talk will delve into today’s research landscape and the future applications of these technologies for first responders, both during training and in the field.  

Tell us about your research as it pertains to emergency response. What are you working on now?

What I focus on in my research is the role of games, simulations, and virtual worlds in connecting people with meaningful experiences – the kinds of education, learning, and growth that such experiences can provide. In that context, I’ve been examining things like emotional resonance with interactive storytelling, virtual worlds and their impact on perception, and similar issues. Though my work doesn’t pertain directly to emergency response, I’m connected to labs and research centers that have employed this to great effect for purposes in and around emergency response as a field. The core of my work is a little bit lower level, focused on connection, communication, and collaboration in virtual spaces.

In your opinion, which AR/VR technologies currently in the marketplace offer the best opportunity to help first responders do their jobs more safely and effectively? 

I’d be very hesitant to say one particular technology is a clear winner – it depends what you want to use it for, what the end goal is of a given application. That will dictate technology selection and implementation.

That said, I have seen some amazing apps and ideas with a whole host of platforms. Some developers focus on broad application, while others provide customized solutions. HITLabNZ, one of the research centers I work with, developed a multi-sensory training simulator for wildfire pilots. My colleague at American University, Dr. Pietroszek [Krzysztof Pietroszek, Founding Director, Institute for IDEAS], is working on applications for virtual health in collaboration with George Washington University.

It’s easy to say modern headsets like the Quest or the Hololens 2 are miles ahead of what we had before, but this is a constantly evolving field and it won’t slow down for a while.

How do you foresee the expansion of VR/AR technology in areas such as emergency response over the next five, 10, even 50 years?

In the short term, we’ve got to make this stuff easier to use and control. It’s getting better but the interfaces are still very clunky. In AR, it’s hard to virtually integrate virtual objects in real-world scenes with the right visual cues (shadows, obscuring objects, depth sorting, etc.). In VR, the number-one issue is control. Joystick and hand tracking technology is miles ahead of where it was a few short years ago, but still needs to travel miles ahead to be easily and seamlessly usable.

Finger tracking will help a lot, gesture recognition, having a better sense of the full body in a physical space, and eventually minimizing or optimizing the amount and type of sensor data we need to pull that off. Someday maybe we’ll even be able to read the stimuli from the brain sent as movement directions and use those as inputs to virtual avatars – who knows? But for now just continuing to iterate and understand more about how we interface with virtual and augmented spaces is key. This is one of the reasons that games are so critical to this work – we learn a lot by watching how players adapt to games and their interfaces and iterate their strategies through and around them.

What challenges will need to be overcome to get us there?

Oh, I mean, tons. TONS. I was at a workshop last year that NIH put on about using VR/AR as a new way to start visualizing cancer research data, because we have lots of researchers now that have almost more data than they know what to do with. More data than they can make sense of. And maybe VR/AR is a way to help with that, to try to make it immersive so a scientist can somewhat literally get inside it, can see it, can track it over time, and find patterns in it.

But it underscores that the biggest challenge, beyond all the tech and all the motion tracking and all the usability, is communication and imagination. Cancer researchers and game developers don’t have, by and large, a lot of shared background or training. They have different vocabularies. They think of different things in different ways. And so a lot of times the hardest thing is figuring out what would be useful to each other, how to play in each other’s worlds, and how to help each other with the kinds of problems that might have an impact.

If agencies are considering incorporating VR/AR tools into their workflows, what are the important features they should consider before committing to a particular piece of tech?

First and foremost, talk to people that have used it before, get some test kits and try it out yourself. Don’t just try the demo kits that ship with the hardware – ask to see the software and simulations that you’d be training on in your particular area! These can vary widely, so a great headset or AR app with bad software will still be bad, while good software can suffer from hardware that can’t keep up.

All of the normal IT adoption issues also apply, of course, like cost, security, integration, lifecycle planning, etc. Probably the best approach is to do the homework on where it’s already been successful, or if it’s brand-new / research implementation, set a long enough runway to really have a proper shot at understanding the technology and its fit within the organization. There is no substitute for having your own people try it and giving them long enough to form real opinions on its efficacy within your organization. 

 The Equipped leadership summit, 'Emerging Technologies for Emergency Response,' will take place on March 14, 2023 at American University's Washington College of Law, from 9 a.m. – 1 p.m. The half-day networking event is free to attend and includes curated roundtable discussions, a keynote presentation, a networking lunch, and hands-on product demonstrations. It’s open to leadership-level first responders as well as government representatives, nonprofit advocates, and academics who work within the field of public safety. Learn more here.