The Problem with Simulations

Introduction:

As a student I can be told about central tendency in statistics and the properties of a normal distribution.  I can memorize the difference between mean, mode, and median.  I might even do well on an exam, if asked to calculate the standard deviation.  I could, by rote, follow the four steps, and produce an accurate number – and yet have no concept of variance, and the significance of samples and of sample sizes and understanding the complete picture of central tendency.  Or I could play with a simple simulation as illustrated on the following website:

http://onlinestatbook.com/stat_sim/sampling_dist/index.html

In this simulation I see the real population distribution and I see the output of mean, median and mode – and standard deviation.  I can see that I need a lot of samples in order to start to see the low frequency outliers from the mean.   Rather than being told something, or memorizing a formula, I get to manipulate numbers and see the story of central tendency and variation play out.

I’ve moved from the lower level memorization of a definition and the lower level performance of a procedure to the higher level conceptual understanding that is so important in the field of statistics.

The problem with the term simulation

Perhaps, my example doesn’t quite measure up to your idea of a simulation. That indeed is one problem associated with the concept of ‘simulation’.  We have one word that describes a wide range of things.  A simulation can be any number of things ranging from this ‘smart’ animation of samples to an immersive virtual world, complete with body suits and head gear.  The Inuit, reportedly  had different words for snow, including “matsaaruti” for wet snow and “pukka” for powdery snow. Instructional designers have one inadequate term for a full range of activities:  the simulation.

But that’s not the only problem.

I’ll set out in this post to outline categories of simulations, champion their value, and help clear away some of the obstacles to their adoption.

The need for higher order strategies

Despite their value, simulations represent a very small percentage of online learning activities.  Many business, medical and engineering programs engage their students in simulations, but the ratio of simulation-based activities to all online learning is small.

More than ten years ago, research at Cornell University (Bell, 2008)  cited a study that places simulations at a ‘relatively small percentage (approximately 2-3%) of the total e-learning industry’.  The study states that the costs of producing simulations is high and the effectiveness of simulations has had mixed reviews.  That’s the heart of the problem. The authors of the study suggest that “instructional designers are left with little guidance on how to develop an effective system because the factors that influence the effectiveness of simulation-based training remain unclear.”  Not a very promising start.

But, in my view, simulations are an important strategy for online instructors.  In order for online learning to have any significant impact on learning performance, we need instructors to be skilled at selecting strategies that promote higher order thinking.  Too much of online learning replicates the worst of the classroom experience, in which students passively receive a lecture.  The interactive portion is resigned to a quiz. There are significant alternatives – but they are not easy to implement.  The simulation, as a strategy, is the most challenging.

Simulations are effective because students enjoy engaging in simulations and being challenged to think.  Instructional designers often prescribe or design simulations to promote higher order thinking that helps students synthesize facts, concepts, principles, rules and procedures.

Educational psychologists recognize the value of simulations to promote cognitive complexity – which is the student’s ability to detect nuances and subtle differences that might impact their decisions or judgement.

In a meta-analysis conducted by Dr. Traci Sitzman at the University of Colorado, computer-based simulation games promoted students’ retention of content, belief in their own capacity to complete the tasks, recall of facts, and their procedural knowledge (Sitzmann, 2011).

But what is a simulation – and how can busy, online learning instructors leverage this strategy?

As mentioned, the term ‘simulation’ covers a broad range of activities – from the very simple, to the very sophisticated.  We’ve all seen the complex training simulators used in space and flight training.  A commercial aircraft simulator can run from ½ million to several million dollars.  Clearly outside of our budget.  We’re also familiar with high fidelity simulations in nursing.  They range from virtual reality systems to high fidelity mannequins.  These are two categories of simulations that require significant investment.  There are other types of simulations, however, that are simpler and affordable.  And they can positively impact every discipline.

A Range of Types

Under this umbrella of simple and affordable, we can include a range of simulation types.  In past articles, I’ve written about interactive case studies.  In interactive case studies, students are presented with a case and some resources. They have to do something as a result such as create a business plan, solve a problem, uncover underlying issues…whatever.  In the past, I contributed to a team working on an interactive case study that involved assessing a student’s eligibility for credit for prior learning.

In decision-making scenarios (a type of interactive case study), a student is placed in a situation, must collect information, make a decision and then evaluate that decision based on the expert answer, which may come in the form of feedback from a coach or from the revealed consequence of the decision.  I’ve written about a decision making scenario that placed the student in Abraham Lincoln’s shoes when southern states were threatening to secede.  As a student, you consult the same advisers who Lincoln consulted.  You make a decision and then contrast that with what Lincoln actually did.  The whole idea behind this decision-making activity came from a professor of history at Tulane University.

Kognito  (https://kognito.com/) produces a wide variety of simulations for different audiences, including mental health professionals and school personnel.

One of their products educates faculty, staff, and students about mental health and suicide prevention.  In their simulations, the company employs a variety of strategies:  users interact in an environment made up virtual characters and virtual settings.  The learners role-play by selecting the most appropriate thing to say in a simulated conversation.  Learners get immediate personalize feedback as they engage in decision making in an interactive case study.

Another type of simulation involves students tweaking the values of parameters and seeing the result graphed.  For example, an Isle Royale simulation has students tweaking the initial number of wolves and moose on an island.  After the simulation is started, students watch the wolf and deer population rise and fall until the populations fall into a pattern.  The InsightMaker site hosts thousands of this type of simulation.

Another popular type of simulation is found in the interactions featured at the University of Colorado. https://phet.colorado.edu/  Student learn concepts by changing parameters.  In learning about Ohms law students can increase or decrease voltage, increase or decrease resistance and then see the resulting amperage.  The display is highly visual with all parts of Ohm’s Law graphically illustrated. Even the equation is illustrated, with parts that grow and shrink in size.

 

Screenshot of LodeStar activity with embedded InsightMaker on SIR modeling: Infectious diseases

A Range of Purpose

Simulations fulfill a range of purposes or functions.  The purposes aren’t mutually exclusive.  Simulations may involve one, several or all of these.

Functional or Procedure simulations help learners perform a function in a given situation.   Software simulations, for example, require learners to perform tasks in the software environment.  Vehicle simulators and high fidelity mannequins require learners to do the right thing at the right time.

Conceptual simulations helps learners view a concept in isolation and, in some cases, change the parameters, see the effect and be able to recognize the concept in action.  For example, in a simulation of predator-prey relationships, students see a unique pattern that always develops regardless of the initial number of predators or the initial number of prey.

Process Oriented simulations often include underlying mathematical models – mathematical representations of a real-world system. ‘What-if’ process simulations ask students to make a change to a process and see its outcome.  Students change inputs and immediately view outputs.

Synthesis Oriented simulations involve learners in gathering information, making observations, recalling key principles, concepts and facts and then putting it all together to make the appropriate choices.   Decision-making and interactive case studies are examples.

Behavior Oriented simulations engage students in the affective domain and require students to choose the appropriate behaviors and demonstrate the right attitude given a situation.  Choosing to recycle garbage or choosing to manage time are examples.

In short, types of simulations align nicely with types of knowledge.  Less important is the technology – virtual world, versus two-dimensional animation, versus text narrative – and more important is the behavioral and cognitive change.

By focusing on what is important and eliminating what is not important, we can pare away cost and remove one of the obstacles to using simulations in our curriculum.

Definition

It is difficult to sum up simulations in a single definition and so I offer these attributes.

An educational simulation:

• Loosely or closely represents reality (low versus high fidelity)
  • Represents or models the behavior or characteristics of a system
  • Mimics the outcomes that happen in the natural world.
  • Pares away unnecessary detail
• Stimulates a response in the learner
• Presents learners with a situation that causes them to think – that is, draw upon their knowledge and procedural and analytical skills to make decisions, to form hypotheses, to draw conclusions, to state rules or act in some way
• Provides feedback

Under this broader definition, a disease model that shows a population that is susceptible to, infected by, and recovered from a disease is a simulation.  It is a particularly useful simulation if its underlying math and logic represents a real world phenomenon – even if it is an over-simplification. It is also useful if it allows the student to change parameters of the model, such as population size, the number who are initially infected, the proximity of members of the population and so forth and then make inferences about the outcome.  In this way, the simulation invites learners to ask ‘what if’ questions.   The results of student input cause learners to think and, perhaps draw their own conclusions about general rules and principles.  Changing parameters and running the simulation provides immediate feedback.

General attributes that make simulations an effective learning strategy

From a meta-analysis (Cook, 2013)  focused on simulations involving virtual worlds, high fidelity mannequins, and even human cadavers, we learn about the positive effects of key learning strategies including:

• range of difficulty
• repetitive practice
• multiple learning strategies
• individualized learning
• feedback
• longer time

In short, students benefit from interactions that vary in difficulty, present opportunities for repeated practice, engage them in different ways, adapt to student performance and confidence level, give them time, and, importantly, provide meaningful feedback.  Those are useful characteristics of any eLearning.

Much of eLearning doesn’t include any of these characteristics — not one!   A lot of eLearning is built on voice over PowerPoints that have been imported into an eLearning authoring tool.  The feedback is limited to a score on a final quiz.  More finessed eLearning comes in the form of talking head videos with chapter quizzes.  Many of the learning platforms that allow instructors to market their courses don’t even bother with the import of interactive learning objects.  They support video and audio files and PDFs – that is, presentation formats, not interaction formats.

By necessity, the corporate world relies on voice-over PowerPoints.  High-end eLearning development shops bristle at the prospect of creating a voice-over PowerPoint.  They are often engaged in making highly creative learning objects that impact a lot of employees and yield a high return on the investment.  When I worked for these companies, we developed six figure learning objects that would reduce service calls, for example, and save a company tens of thousands of dollars or cut down on the use of natural gas, to cite another example,  and save a utility tens of thousands of dollars.  But the economics don’t always support such high-cost investments.  The continuing education industry for medical and accounting professionals, for example, is characterized by literally thousands of voice-over PowerPoints.  These industries change so fast.  The demand far outpaces our ability to create quality learning experiences.

Instructors may recognize or accept that simulations are important, but don’t know where to begin. Obviously, building a half-million dollar simulator is out-of-reach, but there is something that instructors can do to make use of this strategy.  The next section is dedicated to some practical suggestions.

Simulation tools

There are a number of web sites that provide free authoring, hosting, and viewing of simulations. One of my favorite cloud-based simulation tools is InsightMaker. (https://insightmaker.com) InsightMaker supports a variety of different simulation types.  Instructors can build their own simulations and models or use one of thousands that have been created across many disciplines. I want to emphasize that last point.  You will be able to find a simulation that you can use – but it may take a little patience and perseverance.

In biology, an instructor can find simulations on food chain, prey/predator population dynamics and much more.  In business, one might find sales forecasting, or marketing simulations.

In ecology, an instructor can simulate the tipping effect of climate change when shrinking icecaps accelerate climate change with bodies of water absorbing radiation rather than reflecting it.   Students can change the values of parameters and see change accelerate.

Here are other sites and examples worth investigating:

https://blog.cathy-moore.com/resources/elearning-samples/

https://phet.colorado.edu/en/simulations/category/chemistry

https://elearning.cpp.edu/learning-objects/organic-chemistry/tlc/

 

And for the engineer:

https://www.youtube.com/watch?v=iOmqgewj5XI

https://www.youtube.com/watch?v=CFwrfoyRE6c

 

Conclusion

There are a number of ways to get started using simulations.  Finding simulation websites is one; finding cloud-based modelling tools is another.

There are a lot of elements to a simulation.   The authors of the Cornell study suggest that all too often we focus on the technology of simulation rather than on the critical educational elements that are found in the content, the level of immersion (fidelity related to the real world), the interaction, and communication.  The cost is strongly associated with the design and the production of content – the imagery, music, the interface, etc.  The interaction, however, may be accomplished relatively inexpensively with text narratives and decision-making (supported by authoring tools).  The last element, communication, can certainly be facilitated through the learning management system discussion board or group discussion in the classroom.  If we can study these elements discretely and evaluate their impact on learning, as instructional designers, we can separate high cost artwork and media production (that may have little instructional value) from low-cost instructional strategies that provide great value in terms of learning outcomes.

References

Bell, B. S., Kanar, A. M. & Kozlowski, S. W. J. (2008). Current issues and future directions in simulation-based training (CAHRS Working Paper #08-13). Ithaca, NY: Cornell University, School of Industrial and Labor Relations, Center for Advanced Human Resource Studies. http://digitalcommons.ilr.cornell.edu/cahrswp/492

Sitzmann, Traci, (2011) A Meta-Analytic Examination of the Instructional Effectiveness of Computer-based simulation games

Cook DA, Hamstra SJ, Brydges R, Zendejas B, Szostek JH, Wang AT, Erwin PJ,
Hatala R. Comparative effectiveness of instructional design features in
simulation-based education: systematic review and meta-analysis. Med Teach.
2013;35(1):e867-98. doi: 10.3109/0142159X.2012.714886. Epub 2012 Sep 3. Review.
PubMed PMID: 22938677.

 

Postscript: A Proposed Low-fidelity, low cost simulation

On a personal note, for the last several years I’ve been thinking about low-cost simulations that pay high dividends in terms of student outcomes.  As mentioned, I’ve written about decision-making scenarios and interactive case studies.

My latest experiment has been with a model that I call a State Response Engine (SRE).  In the future I hope to write extensively about it.  Briefly, SRE presents the learner with a randomized state and requires the appropriate response.

To better understand SRE, let’s imagine this eLearning activity.  The learner is an online instructor.  The situation is that the college dean has presented the instructor with a set of learning goals.  The online instructor must follow the appropriate process in order to select, develop and evaluate activities and assessments that will align to the goal and help students achieve that goal.

The random state comes in the form of a specific student audience and learning goal.  The engine (the computer program) selects an audience (e.g. non-majors versus majors or freshman versus capstone students, etc.)  From that point forward all of learner responses and any future random states relate to the first choice.  If the computer chose senior students completing their capstone – all of the future states relate to senior students.  All of the resources that appear relate to senior students.  The learners can then investigate the resources for key situational factors.  The engine then randomly selects a learning goal.  The goal might involve the capstone students in promoting conceptual knowledge or putting it all together – but a goal of recalling some basic facts and figures would not be in the selection pool.

The engine then displays resources connected to the state and options in the form of learner responses.  Some of the options or choices would be valid regardless of the goal and student audience.  Others would be valid only for a specific type of knowledge or a class of learner.

The learner progresses through phases or categories.  The phases might be specific stages in a process or something else.  In this case, the phases relate to recognizing situational factors, developing objectives, designing assessments, and designing activities.  In short, a backward design process.  Some of the response options will be correct; others will be incorrect based on the randomly chosen state.   At every stage, learners will be shown links to resources that will help them make the right decisions.  After learners have chosen what they judged to be the right responses, they submit for evaluation.  They then receive a response by response critique and an overall score.

That’s it in a nutshell.  It may or may not be a useful arrow in the instructor’s quiver, but we must continue to search for low-cost high-yield strategies that promote higher-order thinking.  I’ll continue in this pursuit and celebrate other attempts to create effective strategies.