Zoom Backgrounds

(To download a background, right-click and “Save image as…”)

Full Workshop Agenda

PDF version available here

All times in Eastern Daylight Time (EDT).

Use of “A / B” in the schedule denotes parallel sessions.

Monday, April 12th

8:45 AM-8:55 AM

Opening Remarks

Session Recording URL:

Norton Schwartz
Institute for Defense Analyses


Abstract: Norton A. Schwartz serves as President of the Institute for Defense Analyses (IDA), a nonprofit corporation operating in the public interest. IDA manages three Federally Funded Research and Development Centers that answer the most challenging U.S. security and science policy questions with objective analysis leveraging extraordinary scientific, technical, and analytic expertise. At IDA, General Schwartz (U.S. Air Force, retired) directs the activities of more than 1,000 scientists and technologists employed by IDA. General Schwartz has a long and prestigious career of service and leadership that spans over 5 decades. He was most recently President and CEO of Business Executives for National Security (BENS). During his 6-year tenure at BENS, he was also a member of IDA’s Board of Trustees. Prior to retiring from the U.S. Air Force, General Schwartz served as the 19th Chief of Staff of the U.S. Air Force from 2008 to 2012. He previously held senior joint positions as Director of the Joint Staff and as the Commander of the U.S. Transportation Command. He began his service as a pilot with the airlift evacuation out of Vietnam in 1975. General Schwartz is a U.S. Air Force Academy graduate and holds a master’s degree in business administration from Central Michigan University. He is also an alumnus of the Armed Forces Staff College and the National War College. He is a member of the Council on Foreign Relations and a 1994 Fellow of Massachusetts Institute of Technology’s Seminar XXI. General Schwartz has been married to Suzie since 1981.

8:55 AM-9:05 AM

Opening Remarks

Session Recording URL:

Raymond O’Toole
DOT&E


Abstract: Dr. O’Toole is the Acting Director, Operational Test and Evaluation as of January 20, 2021. Dr. O’Toole was appointed as the Principal Deputy Director, Operational Test and Evaluation in February 2020. In this capacity he is the principal staff assistant for all functional areas assigned to the office. He participates in the formulation, development, advocacy, and oversight of policies of the Secretary of Defense and in the development and implementation of test and test resource programs. He supports the Director in the planning, conduct, evaluation and reporting of operational and live fire testing. He serves as the Appropriation Director and Comptroller for the Operational Test and Evaluation, Defense Appropriation and the principal advisor to the Director on all Planning, Programming, and Budgeting System matters. Dr. O’Toole is the former Deputy Director for Naval Warfare within DOT&E. He oversaw the operational and live-fire testing of ships and submarines and their associated sensors; combat and communications systems, and weapons. He was also responsible for overseeing the adequacy of the test infrastructure and resources to support operational and live-fire testing for all acquisition programs across the Defense Department. Dr. O’Toole was previously an employee of the Naval Sea Systems Command as the Deputy Group Director of Aircraft Carrier Design and Systems Engineering. Prior to that, he was the Director of Systems Engineering Division (Submarines and Undersea Systems) where he led a diverse team of engineers who supported all Submarine Program Managers. His other assignments include being a Ship Design Manager/Navy’s Technical Authority for the USS VIRGINIA Class submarines during design and new construction and for Amphibious Ships, Auxiliary Ships, and Command & Control Ships during inservice operations. Dr. O’Toole has also held other positions within the Department of Defense such as Deputy Program Executive Officer (Maritime and Rotary Wing) at the United States Special Operations Acquisition Command, Staff to the Deputy Assistant Secretary of the Navy for Research, Development & Acquisition (Ship Programs), and Deputy Director of Regional Maintenance for COMPACFLT (N43). In addition, Dr. O’Toole has over 30 years of experience as a Naval Officer (Active and Reserve) retiring at the rank of CAPTAIN. His significant tours include 5 Commanding Officer tours. Dr. Raymond D. O’Toole, Jr. is a native of Long Island NY and a graduate of the State University of New York – Maritime College earning a Bachelor of Engineering in Marine Engineering. He also holds a Master of Engineering Degree in Systems Engineering from Virginia Polytechnic Institute and State University, a Master of Science Degree in National Resource Strategy from the Industrial College of the Armed Forces, and a Doctorate in Engineering in the field of Engineering Management from the George Washington University, where he is now a Professional Lecturer of Engineering Management and Systems Engineering. He has received the SECDEF Meritorious Civilian Service Award and the USN Meritorious and Superior Civilian Service Awards.

9:00 AM-9:45 AM

Keynote

Assessing Human-Autonomy Interaction in Driving-Assist Settings

Session Materials:

Session Materials URL:

Session Recording URL:

Mary “Missy” Cummings
Duke University


Abstract: In order to determine how the perception, Autopilot, and driver monitoring systems of Tesla Model 3s interact with one another, and also to determine the scale of between- and within-car variability, a series of four on-road tests were conducted. Three sets of tests were conducted on a closed track and one was conducted on a public highway. Results show wide variability across and within three Tesla Model 3s, with excellent performance in some cases but also likely catastrophic performance in others. This presentation will not only highlight how such interactions can be tested, but also how results can inform requirements and designs of future autonomous systems.

Bio: Professor Mary (Missy) Cummings received her B.S. in Mathematics from the US Naval Academy in 1988, her M.S. in Space Systems Engineering from the Naval Postgraduate School in 1994, and her Ph.D. in Systems Engineering from the University of Virginia in 2004. A naval pilot from 1988-1999, she was one of the U.S. Navy’s first female fighter pilots. She is currently a Professor in the Duke University Electrical and Computer Engineering Department and the Director of the Humans and Autonomy Laboratory. She is an AIAA Fellow and a member of the Veoneer, Inc. Board of Directors

9:45 AM-10:00 AM

Break

10:00 AM-11:30 AM

Session 1A

Theme: Modeling & Simulation
Tom Johnson, Session Chair


Advancements in Characterizing Warhead Fragmentation Events

Session Materials:

Session Recording URL:

John Haman
Institute for Defense Analyses

Modeling and Simulation in Support of the Decision Analysis Process

Session Materials:

Session Recording URL:

Michael Greco
U.S. Army CCDC Armaments Center

Uncertainty Quantification and Sensitivity Analysis Methodology for AJEM
Craig Andres
U.S. Army CCDC Data & Analysis Center

Session 1B

Theme: Integrated Testing
Joe Warfield, Session Chair


A Framework for Efficient Operational Testing through Bayesian Adaptive Design

Session Materials:

Victoria Sieck
University of New Mexico / Air Force Institute of Technology

A Great Test Requires a Great Plan

Session Materials:

Session Recording URL:

Aaron Ramert
Scientific Test and Analytics Techniques Center of Excellence (STAT COE)

Prior Formulation in a Bayesian Analysis of Biomechanical Data

Session Materials:

Session Recording URL:

Amanda French
Johns Hopkins University Applied Physics Laboratory

11:30 AM-12:00 PM

Break

12:00 PM-1:00 PM

Roundtables *Pre-Registration Required*

Roundtable Discussion

Theme: Special Topics


Opportunities and Challenges for Openly Publishing Statistics Research for National Defense

Session Materials:

Lyndsay Shand
Sandia National Laboratories

Roundtable Discussion

Theme: Special Topics


Overcoming Challenges and Applying Sequential Procedures to T&E
Rebecca Medlin
Institute for Defense Analyses

1:00 PM-2:30 PM

Mini Tutorial 1A

Theme: Modeling & Simulation
Sarah Burke, Session Chair


Statistical Approaches to V&V and Adaptive Sampling in M&S – Part 1

Session Materials:

Session Recording URL:

Jim Wisnowski
Adsurgo LLC

Statistical Approaches to V&V and Adaptive Sampling in M&S – Part 2

Session Materials:

Jim Simpson
JK Analytics

Mini Tutorial 1B

Theme: Modeling Human-System Interaction
Rebecca Medlin, Session Chair


Introduction to Qualitative Methods – Part 1

Session Materials:

Session Recording URL:

Kristina Carter
Institute for Defense Analyses

Introduction to Qualitative Methods – Part 2
Daniel Hellman
Institute for Defense Analyses

Introduction to Qualitative Methods – Part 3
Emily Fedele
Institute for Defense Analyses

2:30 PM-2:45 PM

Break

2:45 PM-4:15 PM

Session 2A

Theme: Special Topics
Angie Patterson, Session Chair


Statistical Engineering in Practice

Session Materials:

Session Recording URL:

Roger Hoerl
Union College

Statistical Engineering in Practice

Session Materials:

Session Recording URL:

Alex Varbanov
Procter and Gamble

Statistical Engineering in Practice

Session Recording URL:

Peter Parker
NASA Langley

Session 2B

Theme: Test & Evaluation Methods for Emerging Technology and Domains
Jason Schlup, Session Chair


Operational Cybersecurity Test and Evaluation of Non-IP and Wireless Networks

Session Materials:

Session Recording URL:

Peter Mancini
Institute for Defense Analyses

Cybersecurity Metrics and Quantification: Problems, Some Results, and Research Directions

Session Materials URL:

Shouhuai Xu
University of Colorado Colorado Springs

Collaborative Human AI Red Teaming

Session Materials:

Galen Mullins
Johns Hopkins University Applied Physics Laboratory

4:15 PM-4:30 PM

Break

4:30 PM-5:30 PM

Session 3A

Theme: Modeling & Simulation
Melissa Rivers, Session Chair


Verification and Validation of Elastodynamic Simulation Software for Aerospace Research

Session Materials:

Erik Frankforter
NASA Langley Research Center

Challenges in Verification and Validation of CFD for Industrial Aerospace Applications

Session Materials:

Andrew Cary
Boeing Research and Technology

Certification by Analysis: A 20-year Vision for Virtual Flight and Engine Testing

Session Materials:

Timothy Mauery
Boeing

Session 3B

Theme: Modeling & Simulation
Han Yi, Session Chair


Estimating Pure-Error from Near Replicates in Design of Experiments

Session Materials:

Caleb King
SAS Institute

Surrogate Models and Sampling Plans for Multi-fidelity Aerodynamic Performance Databases

Session Materials:

Kevin Quinlan
Lawrence Livermore National Laboratory

Tuesday, April 13th

8:45 AM-9:45 AM

Session 4A

Theme: Modeling & Simulation
Ying Shi, Session Chair


Physics-Informed Deep Learning for Modeling and Simulation under Uncertainty

Session Materials:

Patrick Leser
NASA Langley Research Center

Fast, Unbiased Uncertainty Propagation with Multi-model Monte Carlo

Session Materials:

Geoffrey Bomarito
NASA Langley Research Center

Session 4B

Theme: Special Topics
Matthew Avery, Session Chair


Empirical Analysis of COVID-19 in U.S. States and Counties

Session Materials:

Emily Heuring
Institute for Defense Analyses

Spatio-Temporal Modeling of Pandemics

Session Materials:

Nicholas Clark
West Point

10:00 AM-11:30 AM

Session 5A

Theme: Modeling & Simulation
Ken Toro, Session Chair


Assessing Next-Gen Spacesuit Reliability: A Probabilistic Analysis Case Study at NASA

Session Materials:

James Warner
NASA Langley Research Center

Entropy-Based Adaptive Design for Contour Finding and Estimating Reliability

Session Materials:

Austin Cole
Virginia Tech

Debunking Stress Rupture Theories Using Weibull Regression Plots

Session Materials:

Anne Driscoll
Virginia Tech

Session 5B

Theme: Test & Evaluation Methods for Emerging Technology and Domains
Jeffery Smith, Session Chair


Intelligent Integration of Limited-Knowledge IoT Services in a Cross-Reality Environment

Session Materials:

Mark Dennison
U.S. Army DEVCOM Army Research Laboratory

Multi-Agent Adaptive Coordinated Autonomy in Contested Battlefields

Session Materials:

John Rogers
U.S. Army DEVCOM Army Research Laboratory

Army’s Open Experimentation Test Range for Internet of Battlefield Things: MSA-DPG

Session Materials:

Jade Freeman
U.S. Army DEVCOM Army Research Laboratory

11:30 AM-12:00 PM

Break

12:00 PM-1:00 PM

Roundtables *Pre-Registration Required*

Roundtable Discussion

Theme: Special Topics


The Role of the Statistics Profession in the DoD’s Current AI Initiative
Laura Freeman
Virginia Tech

Roundtable Discussion

Theme: Special Topics


Identifying Challenges and Solutions to T&E of Non-IP Networks
Peter Mancini
Institute for Defense Analyses

1:00 PM-2:30 PM

Mini Tutorial 2A

Theme: Modeling Human-System Interaction
Tom Donnelly, Session Chair


Introduction to Structural Equation Modeling: Implications for Human-System Interactions

Session Materials:

Session Recording URL:

Laura Castro-Schilo
SAS Institute

Mini Tutorial 2B

Theme: Test & Evaluation Methods for Emerging Technology and Domains
Lance Fiondella, Session Chair


Combinatorial Interaction Testing

Session Materials URL:

Session Recording URL:

Erin Lanus
Virginia Tech

2:30 PM-2:45 PM

Break

2:45 PM-4:00 PM

Panel

Finding the Human in the Loop: Evaluating Warfighters’ Ability to Employ AI Capabilities

Session Recording URL:

Dan Porter
Institute for Defense Analyses

Finding the Human in the Loop: HSI | Trustworthy AI

Session Recording URL:

Stoney Trent
Virginia Tech; The Bulls Run Group, LLC

Finding the Human in the Loop: Considerations for AI in Decision Making

Session Materials:

Session Recording URL:

Joe Lyons
711 Human Performance Wing at Wright-Patterson AFB

Finding the Human in the Loop: Evaluating HSI with AI-Enabled Systems: What should you consider in a TEMP?

Session Materials:

Session Recording URL:

Jane Pinelis
Department of Defense Joint Artificial Intelligence Center (JAIC)

Finding the Human in the Loop: Panelist

Session Recording URL:

Chad Bieber
Johns Hopkins University Applied Physics Laboratory

Finding the Human in the Loop: Panelist

Session Recording URL:

Poornima Madhavan
MITRE

Finding the Human in the Loop: Panelist

Session Recording URL:

Rachel Haga
Institute for Defense Analyses

4:00 PM-4:15 PM

Presentation of Army Wilks Memorial Award

Session Materials URL:

Winner To Be Announced

Wednesday, April 14th

8:45 AM-9:45 AM

Session 6A

Theme: Test & Evaluation Methods for Emerging Technology and Domains
Curtis Miller, Session Chair


A Metrics-based Software Tool to Guide Test Activity Allocation

Session Materials:

Jacob Aubertine
University of Massachusetts Dartmouth

An Adaptive Approach to Shock Train Detection

Session Materials:

Greg Hunt
William & Mary

Session 6B

Theme: Modeling & Simulation
Kevin Singer, Session Chair


A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge

Session Materials:

Melissa Jablonski
US Army CCDC Armaments Center

A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge

Session Materials:

Sarah Longo
US Army CCDC Armaments Center

9:45 AM-10:00 AM

Break

10:00 AM-11:30 AM

Session 7A

Theme: Modeling Human-System Interaction
Rachel Haga, Session Chair


Characterizing Human-Machine Teaming Metrics for Test and Evaluation

Session Materials:

Brian Vickers
Institute for Defense Analyses

Automated Test Case Generation for Human-Machine Interaction

Session Materials:

Matthew Bolton
University at Buffalo, the State University of New York

Cognitive Work Analysis – From System Requirements to Validation and Verification

Session Materials:

Matthew Miller
Jacobs/NASA Johnson Space Center

Session 7B

Theme: Special Topics
Shelley Cazares, Session Chair


Dashboard for Equipment Failure Reports

Session Materials:

Robert Cole Molloy
Johns Hopkins University Applied Physics Laboratory

Metrics for Assessing Underwater Demonstrations for Detection and Classification of UXO

Session Materials:

Jacob Bartel
Institute for Defense Analyses

Machine Learning Reveals that the Russian IRA’s Twitter Topic Patterns Evolved over Time

Session Materials:

Emily Parrish
Institute for Defense Analyses

11:30 AM-12:00 PM

Break

12:00 PM-1:00 PM

Roundtables *Pre-Registration Required*

Roundtable Discussion

Theme: Special Topics


Organizing and Sharing Data within the T&E Community
Matthew Avery
Institute for Defense Analyses

Roundtable Discussion

Theme: Special Topics


Test Design and Analysis for Modeling & Simulation Validation
Kelly Avery
Institute for Defense Analyses

1:00 PM-2:30 PM

Mini Tutorial 3A

Theme: Test & Evaluation Methods for Emerging Technology and Domains
Tom Donnelly, Session Chair


Pseudo-Exhaustive Testing – Part 1

Session Materials:

Session Materials URL:

Session Recording URL:

Ryan Lekivetz
SAS Institute

Pseudo-Exhaustive Testing – Part 2

Session Materials:

Session Recording URL:

Joseph Morgan
SAS Institute

Mini Tutorial 3B

Theme: Test & Evaluation Methods for Emerging Technology and Domains
Doug Ray, Session Chair


Introduction to Neural Networks for Deep Learning with Tensorflow

Session Materials URL:

Session Recording URL:

Roshan Patel
US Army CCDC Armaments Center

2:30 PM-2:45 PM

Break

2:45 PM-4:00 PM

Panel

The Keys to Successful Collaborations during Test and Evaluation: Moderator

Session Recording URL:

Christine Anderson-Cook
Los Alamos National Lab

The Keys to Successful Collaborations during Test and Evaluation: Panelist

Session Materials:

Session Recording URL:

Sarah Burke
STAT Center of Excellence

The Keys to Successful Collaborations during Test and Evaluation: Panelist

Session Recording URL:

Willis Jensen
W.L. Gore & Associates

The Keys to Successful Collaborations during Test and Evaluation: Panelist

Session Recording URL:

John Haman
Institute for Defense Analyses

The Keys to Successful Collaborations during Test and Evaluation: Panelist

Session Recording URL:

Peter Parker
NASA Langley

4:00 PM-4:10 PM

Closing Remarks

Session Recording URL:

William “Allen” Kilgore
NASA Langley Research Center


Abstract: Mr. William (Allen) Kilgore serves as Director, Research Directorate at NASA Langley Research Center. He previously served as Deputy Director of Aerosciences providing executive leadership and oversight for the Center’s Aerosciences fundamental and applied research and technology capabilities with the responsibility over Aeroscience experimental and computational research. After being appointed to the Senior Executive Service (SES) in 2013, Mr. Kilgore served as the Deputy Director, Facilities and Laboratory Operations in the Research Directorate. Prior to this position, Mr. Kilgore spent over twenty years in the operations of NASA Langley’s major aerospace research facilities including budget formulation and execution, maintenance, strategic investments, workforce planning and development, facility advocacy, and integration of facilities’ schedules. During his time at Langley, he has worked in nearly all of the major wind tunnels with a primary focus on process controls, operations and testing techniques supporting aerosciences research. For several years, Mr. Kilgore led the National Transonic Facility, the world’s largest cryogenic wind tunnel. Mr. Kilgore has been at NASA Langley Research Center since 1989, starting as a graduate student. Mr. Kilgore earned a B.S. and M.S. in Mechanical Engineering with concentration in dynamics and controls from Old Dominion University in 1984 and 1989, respectively. He is the recipient of NASA’s Exceptional Engineering Achievement Medal in 2008 and Exceptional Service Medal in 2012.