In the shadows of consumer-accessible augmented reality, a Defense Advanced Research Projects Agency (DARPA) training program known as Accelerated Learning has laid the groundwork for reinventing how we train our military–with virtual reality. In the early 2000s, DARPA conducted neuroscience research to build cognitive metrics tracking brain performance for attention, distraction, and drowsiness during military simulations. The results of this research are beginning to produce technological innovations that could forever change how we train our military and play our video games.
One of these innovations is called NeuroBridge, a software tool built by Intific, Inc. (acquired by Cubic in 2014) and headed by former DARPA Program Manager, and current researcher and neuroscientist, Dr. Amy Kruse. Kruse joined Intific in 2010 as the VP of Innovation, five years after Intific started building NeuroBridge. Her goal was to use the expertise of software developers to connect cutting-edge neuroscience with real-life simulations, operations, training, and even gaming.
“When I was leaving DARPA, one of the things I really wanted to do was take the industry and the technology to the next level. We did a lot of great research on ways of connecting individuals using real-time measures of brain activity, physiological activity, and actually connecting those to real operational applications,” said Kruse.
In this way, NeuroBridge is much more user-focused software. It has the ability to connect the physiological and neurophysiological activity of a player to live simulation and gaming environments. Kruse says this two-way function (or closed loop) allows communication between the sensors and the simulation. More specifically, this functionality lets players actually control aspects of simulations with their mind, or cognitive state.
“So if there’s a particular state, event, or aspect of the individual; what they’re experiencing can actually drive the simulation,” said Kruse.
To better illustrate this, Intific points to their game NeuroStorm, which was originally created at the 2012 Google I/O after hours event. It is a simple “shooter on rails” arcade game, with the premise of floating through a brain in a nanobot, armed with a laser, through a sea of brain neurons. The objective is for players to zap and charge neurons as fast and accurately as possible, in order to score more points and increase the effectiveness of their shooting. Because Intific has integrated NeuroBridge with this simulation, players can now actually control aspects of the game mechanics with their EEG brain activity.
NeuroBridge was connected to NeuroStorm by adding a “focus level” bar to the game, which directly synchronizes with a player’s brain activity and live gameplay. Being a good shot is not enough to produce enough “focus” to win, so the longer a player can stay in the zone, the stronger the laser will become, offering bonuses during gameplay and higher scores at the end.
“It can be used for research purposes in correlation, but also a closed loop for entertainment purposes,” Kruse explained.
Gaming the American Sniper
Some of Kruse’s past military research was based on these exact cognitive focus training principles.
“It turns out expert marksman have a very particular brain state they can enter at will, allowing them to get positioned, control breathing, heart rate, and brain activity,” said Kruse.
DARPA’s simulated training for military snipers uses this information to train novice shooters to alter their cognitive state to match and enter this specific cognitive state. If a sniper shows all of these physiological and neurophysiological things at the same time, they will almost always take an expert shot.
“You would find that their performance improved (not instantly) fairly substantially without them even having to sit on the range and take shots. Literally by training their brains, they improved their performance.”
In fact, the research had proven so useful, Cubic and Intific have now integrated NeuroBridge into a virtual marksmanship training system for one of its Department of Defence (DoD) contracts. It’s currently being used globally to train military members with various arms in a stand-up VR-environment with mock weapons.
NeuroBridge Meets VR
The NeuroStorm game was first built for use with a controller, but with the integration of NeuroBridge, it now being paired with state-of-the-art headset and VR technology.
“We’ve also integrated experiences with the Oculus Rift. The same NeuroStorm game, we’ve adapted it to be played completely with the Oculus Rift, with the NeuroBridge and the neuroscience feeding the same role in the game,” said Kruse.
In addition, a mobile version of NeuroBridge has been integrated with Google Glass, allowing real time feedback with Google’s headgear. Most of this work is for basic neuroscience research.
“We are on several other contracts at DARPA looking at novel user interfaces and user experiences in the context of cyber operations. We have used Google glass, Oculus Rift, and some other augmented reality devices; lots of ways of thinking about how humans interact with that type of data in a novel way.”
“NeuroBridge is Hardware Agnostic.”
It’s important to note that Intific only designs NeuroBridge as software, and because of that, it’s much more flexible. Kruse says NeuroBridge can work with any physiological or neurophysiological sensor.
“We design NeuroBridge software to be hardware agnostic. You need some hardware, but it’s not like it’s tied to one particular format. We’ve worked on everything from very high quality research EEG, all the way down to things that are, what I consider, commercial or consumer neuroscience facing (i.e. NeuroSky). We can play along the whole spectrum,” said Kruse.
While EEG is necessary to measure brain activity, Kruse suggest developers could connect NeuroBridge to a game and do similar types of closed loop interactions, using measurements like heart rate and Galvanic Skin Response with fitness wearables.
Age of the Wearables
Keeping an open source mind, Intific also allows developers to tinker and build applications with their Neurobridge 2.0 Beta, which is free to download on their website for researchers and students. They sell the software licensing for commercial applications as well. The Beta comes bundled with the NeuroBridge’s SDK, which allows hardware integration of a developer’s choice.
In its current status, Kruse thinks the NeuroBridge is at the beginning of something much bigger. And she’s right. NeuroBridge has snagged Intific 10 contracts over the past few years. And the majority come with a hefty sum from DoD for use in military training or cognitive skill development. And outside of military use, transportation, commercial and non-profit agencies are looking to add NeuroBridge to existing applications.
NeuroBridge is also becoming well-known in the industry, having been demoed at some of the most highly-acclaimed conferences and tradeshows including Google I/O; Society for Neuroscience; NeuroGaming: ESCONS; I/ITSEC; SOFIC; Serious Play; Department of Education STEM events.
Kruse says she likes to compare the current status of neuroscience tech to pedometers from decades past.
“It just counted steps. Now, something simplistic has blossomed into a whole field with apps, and hardware, and everything else. I really think we’re at the very beginning of that space for neuroscience. Once we start to build and blossom the software for that space, and get it from a clunky pedometer to something awesome that you wear…that’s the same trajectory we’re on. Certainly in the commercial space,” said Kruse.