Intelligent Scenario Management Framework (ISMF)

In the NAVAIR-sponsored Intelligent Scenario Management Framework (ISMF) small-business innovation research (SBIR) initiative, KBSI developed a robust method and a set of support tools for automating the generation and management of training scenarios in response to training goals and student performance.  Specifically, student deficiencies are identified through the use of Bayesian models and evaluation scripts, the visualization and annotation of simulation trace data, and through data analysis and data mining techniques.

Phase II Development

The Phase II effort resulted in a robust scenario generation methodology and an application framework that consists of seven major functions:

• Student Profiler for recording student performance in key areas of competence;
• Scenario Planner to configure scenarios and their key parameters;
• Scenario Execution in which the simulation trace from student execution against the scenario is collected;
• Simulation Trace Interpreter, an intelligent “trace-walker,” it both interprets the trace results at multiple levels to determine student actions and intents;
• Simulation Trace Analyzer in which evaluation scripts are executed to evaluate students’ decision-making abilities;
• Simulation Trace Visualization in which simulation traces are visualized and annotated; and
• Automatic Scenario Generator, in which a Bayesian network model is developed that encodes subject-matter experts’ subjective grading criteria for making overall performance assessments.  The scores assigned to the student for each key area of competence are used as a basis to recommend new initial-condition (IC) set for subsequent training.

Two pilot demonstrations were used to validate the ISMF methodology.  The first demonstration illustrated the management of training scenarios and Initial Condition (IC) sets, archiving of student training history, initialization of IC sets for training, and configuration of IC sets.  The second demonstration illustrated automated scenario generation for a two-on-two, defensive-counter-air engagement (2V2 DCA) scenario.  The simulation was performed with the Joint Semi-Automated Forces (JSAF) v3.1 platform.  Pilot skills such as airspace management, battle management, weapons implementation management, targeting, and weapons engagement zone management were evaluated using ISMF.  These demonstrations confirmed the viability of automated scenario generation to reduce the cognitive load on instructors.

The results of this research effort demonstrate an approach to reducing Navy fighter-tactics instructors’ cognitive workload.  Training scenarios and IC sets are automatically generated in response to students’ deficiencies in a manner that is pedagogically well grounded.  By using data analysis, data mining, and Bayesian network techniques to analyze student performance and to identify strengths and deficiencies, new scenarios and IC sets are developed that improve student performance in response to training objectives.  This approach makes the scenario-based training paradigm more effective for the Navy.

The ISMF technology will result in the following direct benefits to the Navy:

• Reduction in time and effort required to tailor scenarios to trainees’ deficiencies;
• Rapid modification and reconfiguration of pedagogically grounded training scenarios;
• Alleviation of instructor overload during the analysis of training results, conduct of performance assessments, and provision of real-time feedback;
• Improvement in training quality as a result of better support to the continuous learning process;
• More rapid training and retraining of staff;
• Reuse of scenarios and models in training and in other associated applications; and
• A framework within which research scientists can experiment with existing and future training models and techniques.