Zoom Backgrounds
(To download a background, right-click and “Save image as…”)
Full Workshop Agenda
All times in Eastern Daylight Time (EDT).
Use of “A / B” in the schedule denotes parallel sessions.
Monday, April 12th
8:45 AM-8:55 AM |
Opening Remarks Norton Schwartz ![]() Abstract: Norton A. Schwartz serves as President of the Institute for Defense Analyses (IDA), a nonprofit corporation operating in the public interest. IDA manages three Federally Funded Research and Development Centers that answer the most challenging U.S. security and science policy questions with objective analysis leveraging extraordinary scientific, technical, and analytic expertise. At IDA, General Schwartz (U.S. Air Force, retired) directs the activities of more than 1,000 scientists and technologists employed by IDA. General Schwartz has a long and prestigious career of service and leadership that spans over 5 decades. He was most recently President and CEO of Business Executives for National Security (BENS). During his 6-year tenure at BENS, he was also a member of IDA’s Board of Trustees. Prior to retiring from the U.S. Air Force, General Schwartz served as the 19th Chief of Staff of the U.S. Air Force from 2008 to 2012. He previously held senior joint positions as Director of the Joint Staff and as the Commander of the U.S. Transportation Command. He began his service as a pilot with the airlift evacuation out of Vietnam in 1975. General Schwartz is a U.S. Air Force Academy graduate and holds a master’s degree in business administration from Central Michigan University. He is also an alumnus of the Armed Forces Staff College and the National War College. He is a member of the Council on Foreign Relations and a 1994 Fellow of Massachusetts Institute of Technology’s Seminar XXI. General Schwartz has been married to Suzie since 1981. |
8:55 AM-9:05 AM |
Opening Remarks Raymond O’Toole ![]() Abstract: Dr. O’Toole is the Acting Director, Operational Test and Evaluation as of January 20, 2021. Dr. O’Toole was appointed as the Principal Deputy Director, Operational Test and Evaluation in February 2020. In this capacity he is the principal staff assistant for all functional areas assigned to the office. He participates in the formulation, development, advocacy, and oversight of policies of the Secretary of Defense and in the development and implementation of test and test resource programs. He supports the Director in the planning, conduct, evaluation and reporting of operational and live fire testing. He serves as the Appropriation Director and Comptroller for the Operational Test and Evaluation, Defense Appropriation and the principal advisor to the Director on all Planning, Programming, and Budgeting System matters. Dr. O’Toole is the former Deputy Director for Naval Warfare within DOT&E. He oversaw the operational and live-fire testing of ships and submarines and their associated sensors; combat and communications systems, and weapons. He was also responsible for overseeing the adequacy of the test infrastructure and resources to support operational and live-fire testing for all acquisition programs across the Defense Department. Dr. O’Toole was previously an employee of the Naval Sea Systems Command as the Deputy Group Director of Aircraft Carrier Design and Systems Engineering. Prior to that, he was the Director of Systems Engineering Division (Submarines and Undersea Systems) where he led a diverse team of engineers who supported all Submarine Program Managers. His other assignments include being a Ship Design Manager/Navy’s Technical Authority for the USS VIRGINIA Class submarines during design and new construction and for Amphibious Ships, Auxiliary Ships, and Command & Control Ships during inservice operations. Dr. O’Toole has also held other positions within the Department of Defense such as Deputy Program Executive Officer (Maritime and Rotary Wing) at the United States Special Operations Acquisition Command, Staff to the Deputy Assistant Secretary of the Navy for Research, Development & Acquisition (Ship Programs), and Deputy Director of Regional Maintenance for COMPACFLT (N43). In addition, Dr. O’Toole has over 30 years of experience as a Naval Officer (Active and Reserve) retiring at the rank of CAPTAIN. His significant tours include 5 Commanding Officer tours. Dr. Raymond D. O’Toole, Jr. is a native of Long Island NY and a graduate of the State University of New York – Maritime College earning a Bachelor of Engineering in Marine Engineering. He also holds a Master of Engineering Degree in Systems Engineering from Virginia Polytechnic Institute and State University, a Master of Science Degree in National Resource Strategy from the Industrial College of the Armed Forces, and a Doctorate in Engineering in the field of Engineering Management from the George Washington University, where he is now a Professional Lecturer of Engineering Management and Systems Engineering. He has received the SECDEF Meritorious Civilian Service Award and the USN Meritorious and Superior Civilian Service Awards. |
9:00 AM-9:45 AM |
Keynote Assessing Human-Autonomy Interaction in Driving-Assist Settings Mary “Missy” Cummings ![]() Abstract: In order to determine how the perception, Autopilot, and driver monitoring systems of Tesla Model 3s interact with one another, and also to determine the scale of between- and within-car variability, a series of four on-road tests were conducted. Three sets of tests were conducted on a closed track and one was conducted on a public highway. Results show wide variability across and within three Tesla Model 3s, with excellent performance in some cases but also likely catastrophic performance in others. This presentation will not only highlight how such interactions can be tested, but also how results can inform requirements and designs of future autonomous systems. Bio: Professor Mary (Missy) Cummings received her B.S. in Mathematics from the US Naval Academy in 1988, her M.S. in Space Systems Engineering from the Naval Postgraduate School in 1994, and her Ph.D. in Systems Engineering from the University of Virginia in 2004. A naval pilot from 1988-1999, she was one of the U.S. Navy’s first female fighter pilots. She is currently a Professor in the Duke University Electrical and Computer Engineering Department and the Director of the Humans and Autonomy Laboratory. She is an AIAA Fellow and a member of the Veoneer, Inc. Board of Directors |
9:45 AM-10:00 AM |
Break |
10:00 AM-11:30 AM |
Session 1A Theme: Modeling & Simulation Advancements in Characterizing Warhead Fragmentation Events Institute for Defense Analyses Modeling and Simulation in Support of the Decision Analysis Process U.S. Army CCDC Armaments Center Uncertainty Quantification and Sensitivity Analysis Methodology for AJEM Session 1B Theme: Integrated Testing A Framework for Efficient Operational Testing through Bayesian Adaptive Design University of New Mexico / Air Force Institute of Technology A Great Test Requires a Great Plan Scientific Test and Analytics Techniques Center of Excellence (STAT COE) Prior Formulation in a Bayesian Analysis of Biomechanical Data Johns Hopkins University Applied Physics Laboratory |
11:30 AM-12:00 PM |
Break |
12:00 PM-1:00 PM |
Roundtables *Pre-Registration Required* |
Roundtable Discussion Theme: Special Topics Opportunities and Challenges for Openly Publishing Statistics Research for National Defense Sandia National Laboratories Roundtable Discussion Theme: Special Topics Overcoming Challenges and Applying Sequential Procedures to T&E | |
1:00 PM-2:30 PM |
Mini Tutorial 1A Theme: Modeling & Simulation Statistical Approaches to V&V and Adaptive Sampling in M&S – Part 1 Adsurgo LLC Statistical Approaches to V&V and Adaptive Sampling in M&S – Part 2 JK Analytics Mini Tutorial 1B Theme: Modeling Human-System Interaction Introduction to Qualitative Methods – Part 1 Institute for Defense Analyses Introduction to Qualitative Methods – Part 2 Introduction to Qualitative Methods – Part 3 |
2:30 PM-2:45 PM |
Break |
2:45 PM-4:15 PM |
Session 2A Theme: Special Topics Statistical Engineering in Practice Union College Statistical Engineering in Practice Procter and Gamble Statistical Engineering in Practice NASA Langley Session 2B Theme: Test & Evaluation Methods for Emerging Technology and Domains Operational Cybersecurity Test and Evaluation of Non-IP and Wireless Networks Institute for Defense Analyses Cybersecurity Metrics and Quantification: Problems, Some Results, and Research Directions University of Colorado Colorado Springs Collaborative Human AI Red Teaming Johns Hopkins University Applied Physics Laboratory |
4:15 PM-4:30 PM |
Break |
4:30 PM-5:30 PM |
Session 3A Theme: Modeling & Simulation Verification and Validation of Elastodynamic Simulation Software for Aerospace Research NASA Langley Research Center Challenges in Verification and Validation of CFD for Industrial Aerospace Applications Boeing Research and Technology Certification by Analysis: A 20-year Vision for Virtual Flight and Engine Testing Boeing Session 3B Theme: Modeling & Simulation Estimating Pure-Error from Near Replicates in Design of Experiments SAS Institute Surrogate Models and Sampling Plans for Multi-fidelity Aerodynamic Performance Databases Lawrence Livermore National Laboratory |
Tuesday, April 13th
8:45 AM-9:45 AM |
Session 4A Theme: Modeling & Simulation Physics-Informed Deep Learning for Modeling and Simulation under Uncertainty NASA Langley Research Center Fast, Unbiased Uncertainty Propagation with Multi-model Monte Carlo NASA Langley Research Center Session 4B Theme: Special Topics Empirical Analysis of COVID-19 in U.S. States and Counties Institute for Defense Analyses Spatio-Temporal Modeling of Pandemics West Point |
10:00 AM-11:30 AM |
Session 5A Theme: Modeling & Simulation Assessing Next-Gen Spacesuit Reliability: A Probabilistic Analysis Case Study at NASA NASA Langley Research Center Entropy-Based Adaptive Design for Contour Finding and Estimating Reliability Virginia Tech Debunking Stress Rupture Theories Using Weibull Regression Plots Virginia Tech Session 5B Theme: Test & Evaluation Methods for Emerging Technology and Domains Intelligent Integration of Limited-Knowledge IoT Services in a Cross-Reality Environment U.S. Army DEVCOM Army Research Laboratory Multi-Agent Adaptive Coordinated Autonomy in Contested Battlefields U.S. Army DEVCOM Army Research Laboratory Army’s Open Experimentation Test Range for Internet of Battlefield Things: MSA-DPG U.S. Army DEVCOM Army Research Laboratory |
11:30 AM-12:00 PM |
Break |
12:00 PM-1:00 PM |
Roundtables *Pre-Registration Required* |
Roundtable Discussion Theme: Special Topics The Role of the Statistics Profession in the DoD’s Current AI Initiative Roundtable Discussion Theme: Special Topics Identifying Challenges and Solutions to T&E of Non-IP Networks | |
1:00 PM-2:30 PM |
Mini Tutorial 2A Theme: Modeling Human-System Interaction Introduction to Structural Equation Modeling: Implications for Human-System Interactions SAS Institute Mini Tutorial 2B Theme: Test & Evaluation Methods for Emerging Technology and Domains Combinatorial Interaction Testing Virginia Tech |
2:30 PM-2:45 PM |
Break |
2:45 PM-4:00 PM |
Panel Finding the Human in the Loop: Evaluating Warfighters’ Ability to Employ AI Capabilities Dan Porter Finding the Human in the Loop: HSI | Trustworthy AI Stoney Trent Finding the Human in the Loop: Considerations for AI in Decision Making Joe Lyons Finding the Human in the Loop: Evaluating HSI with AI-Enabled Systems: What should you consider in a TEMP? Jane Pinelis Finding the Human in the Loop: Panelist Chad Bieber Finding the Human in the Loop: Panelist Poornima Madhavan Finding the Human in the Loop: Panelist Rachel Haga |
4:00 PM-4:15 PM |
Presentation of Army Wilks Memorial Award Winner To Be Announced |
Wednesday, April 14th
8:45 AM-9:45 AM |
Session 6A Theme: Test & Evaluation Methods for Emerging Technology and Domains A Metrics-based Software Tool to Guide Test Activity Allocation University of Massachusetts Dartmouth An Adaptive Approach to Shock Train Detection William & Mary Session 6B Theme: Modeling & Simulation A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge US Army CCDC Armaments Center A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge US Army CCDC Armaments Center |
9:45 AM-10:00 AM |
Break |
10:00 AM-11:30 AM |
Session 7A Theme: Modeling Human-System Interaction Characterizing Human-Machine Teaming Metrics for Test and Evaluation Institute for Defense Analyses Automated Test Case Generation for Human-Machine Interaction University at Buffalo, the State University of New York Cognitive Work Analysis – From System Requirements to Validation and Verification Jacobs/NASA Johnson Space Center Session 7B Theme: Special Topics Dashboard for Equipment Failure Reports Johns Hopkins University Applied Physics Laboratory Metrics for Assessing Underwater Demonstrations for Detection and Classification of UXO Institute for Defense Analyses Machine Learning Reveals that the Russian IRA’s Twitter Topic Patterns Evolved over Time Institute for Defense Analyses |
11:30 AM-12:00 PM |
Break |
12:00 PM-1:00 PM |
Roundtables *Pre-Registration Required* |
Roundtable Discussion Theme: Special Topics Organizing and Sharing Data within the T&E Community Roundtable Discussion Theme: Special Topics Test Design and Analysis for Modeling & Simulation Validation | |
1:00 PM-2:30 PM |
Mini Tutorial 3A Theme: Test & Evaluation Methods for Emerging Technology and Domains Pseudo-Exhaustive Testing – Part 1 SAS Institute Pseudo-Exhaustive Testing – Part 2 SAS Institute Mini Tutorial 3B Theme: Test & Evaluation Methods for Emerging Technology and Domains Introduction to Neural Networks for Deep Learning with Tensorflow US Army CCDC Armaments Center |
2:30 PM-2:45 PM |
Break |
2:45 PM-4:00 PM |
Panel The Keys to Successful Collaborations during Test and Evaluation: Moderator Christine Anderson-Cook The Keys to Successful Collaborations during Test and Evaluation: Panelist Sarah Burke The Keys to Successful Collaborations during Test and Evaluation: Panelist Willis Jensen The Keys to Successful Collaborations during Test and Evaluation: Panelist John Haman The Keys to Successful Collaborations during Test and Evaluation: Panelist Peter Parker |
4:00 PM-4:10 PM |
Closing Remarks William “Allen” Kilgore ![]() Abstract: Mr. William (Allen) Kilgore serves as Director, Research Directorate at NASA Langley Research Center. He previously served as Deputy Director of Aerosciences providing executive leadership and oversight for the Center’s Aerosciences fundamental and applied research and technology capabilities with the responsibility over Aeroscience experimental and computational research. After being appointed to the Senior Executive Service (SES) in 2013, Mr. Kilgore served as the Deputy Director, Facilities and Laboratory Operations in the Research Directorate. Prior to this position, Mr. Kilgore spent over twenty years in the operations of NASA Langley’s major aerospace research facilities including budget formulation and execution, maintenance, strategic investments, workforce planning and development, facility advocacy, and integration of facilities’ schedules. During his time at Langley, he has worked in nearly all of the major wind tunnels with a primary focus on process controls, operations and testing techniques supporting aerosciences research. For several years, Mr. Kilgore led the National Transonic Facility, the world’s largest cryogenic wind tunnel. Mr. Kilgore has been at NASA Langley Research Center since 1989, starting as a graduate student. Mr. Kilgore earned a B.S. and M.S. in Mechanical Engineering with concentration in dynamics and controls from Old Dominion University in 1984 and 1989, respectively. He is the recipient of NASA’s Exceptional Engineering Achievement Medal in 2008 and Exceptional Service Medal in 2012. |