Game-Based Experimentation for Research in Command and Control and Shared Situational Awareness

Summary

This paper documents the support CNA provided to the Warfare Analysis and Research Department of the Naval War College for an experiment conducted in the spring of 2002. This experiment took the form of a series of games played by teams from the U.S. Naval Academy, the U.S. Air Force Academy, and the Naval War College. The test bed was the internet-based game SCUDHunt, developed earlier by CNA and ThoughtLink Inc. for the Defense Advanced Research Projects Agency. In this simple yet elegant game, players take the roles of sensor asset managers and attempt to deploy their sensors to search a small, gridded map for hidden “SCUD” launchers. Each sensor has different characteristics of coverage and reliability. To play effectively, the players must work together, sharing information and developing their shared situational awareness in order to find the SCUDs and make accurate strike recommendations.

The Naval War College was interested in carrying out this experiment partly as a proof-of-concept about the value of using purpose-built games, and partly as a means of deriving insights into the effects of command styles and visualization techniques. Some of the incentives for this research included providing scientific advice to assist the designers of the Naval War College’s Global War Game series.

The Naval War College has at least two broad concerns related to the Global series—the representation of modern command and control systems and techniques, and the role of shared situational awareness in the concepts underlying them. These concerns intersect strongly with an increased emphasis within the Department of Defense as a whole on how to develop new approaches and methods for Joint command and control that will better take advantage of new technology and concepts—like network-centric warfare and effects-based operations. Indeed, the key future-looking concept for the U.S. military—Joint Vision 2020—includes Joint command and control as one of the key Joint mission areas U.S. forces will have to perform.

Evolving thought in DoD’s command-and-control community emphasizes that improved techniques for creating and maintaining shared situational awareness are key elements of any future system of Joint command and control. U.S. forces will achieve superiority over an adversary by achieving self-synchronization, partly because of increased shared situational awareness. This self-synchronization will enable U.S. forces to act at an increased operational tempo. Improving our understanding of the variables that affect shared situational awareness is critical to implementing these future concepts.

CNA’s previous work for the Defense Advanced Research Projects Agency took some pioneering steps in exploring shared situational awareness using a game-based experimental technique. The Naval War College decided to use a similar approach for this experiment. The earlier CNA study focused on the effects of communications modes and shared-visualization tools on developing shared situational awareness. The Naval War College project focused on command styles and on the use of a “push” visualization mechanism. Measures of interest included a score associated with the shared situational awareness developed by the teams during the course of play, and a score for measuring the accuracy of the estimates of target locations made by the team’s members.

The preliminary results of the analysis of the experimental data proved of only limited interest. Because of some technical and logistical difficulties associated with the execution of the experiment, it is difficult to state with confidence the reliability of the experimental outcomes. That said, the experimental data do seem to hint at a statistically significant improvement in both the shared situational awareness and accuracy of teams employing a “command by direction” style when compared to the same teams playing under “command by influence” or “command by plan” styles. These results indicate potential value in exploring this effect through additional follow-on experiments targeted specifically to explore this factor.

In addition to assisting them with the design and preliminary analysis of their experiment, the Naval War College tasked CNA to consider broader issues related to experimentation in this field.

There are three types of experiments:

1. Experiments designed to explore new ideas or phenomena
2. Experiments designed to test hypotheses
3. Experiments designed to demonstrate new concepts and their feasibility (or lack thereof)

Games provide a wealth of flexibility for exploring, testing, and demonstrating a host of variables and issues associated with decisionmaking. Unfortunately, a single iteration of a complex, multiplayer, largescale operational wargame is expensive in time and money. Such games are poor vehicles for scientific experimentation, for hypothesis testing and “scientific proof.”

The original SCUDHunt game and experiment, as well as the current effort, demonstrate the potential for using a somewhat different sort of gaming environment to formulate and test hypotheses using rigorous scientific and statistical techniques. We can characterize these sorts of games as “distillations”—distinguishing them from simple “abstractions,” like chess, and detailed “simulations,” like Global.

Game-based experimentation is a scientifically rigorous approach to exploring fundamental command-and-control issues. The DoD command- and-control community should increase the use of distillation-style games as part of a program of experimentation and research related to shared situational awareness and command and control. To do so, DoD should look for opportunities to create game-based research efforts—laboratories, if you will—that the paper describes in more detail. Such laboratories are less specific facilities than they are assemblages of critical components and expertise—the games to serve as experimental test beds; the game designers to create the games; and the analysts and scientists to formulate the problems, design the experiments, and analyze the data. Such laboratories could be “virtual” organizations, bringing together subject-matter experts from across the United States and other nations as well. A “virtual, distributed laboratory for game-based experimentation” can help advance our understanding of command and control, information operations, network-centric warfare, and other critical concepts that, in many ways, remain buzzwords rather than realities.

*Originally published in November 2005, this paper was reprinted in December 2024.