Abstracts

John (Jack) R. Boulet, PhD

Vice President
Research and Data Resources
Educational Commission for Foreign Medical Graduates

Abstracts:

• Plenary 6: Simulations to Improve Patient Safety: Challenges and Opportunities
  The use of simulation-based assessments in medicine is expanding at a rapid pace.  This growth has involved numerous modalities, including computer-based case simulations, standardized patients, part-task trainers, and electromechanical mannequins.   While the development of new educational programs and related assessment activities can lead to better trained practitioners, there are numerous challenges, including those associated with finances, assessment administration logistics, scenario development, scoring, standard setting, and the provision of meaningful feedback to trainees.  Most important, gathering evidence to establish the link between simulation-based educational activities and patient safety can be very difficult, often demanding the design and completion of complex longitudinal research studies.  This presentation will provide an overview of simulation-based assessment, concentrating on methods currently employed in the training and evaluation of medical students and residents.   The review of the application of simulation techniques in this arena can provide guidance on how to deal with many of the challenges, especially those associated with the provision of valid and reliable scores.   Knowing more about these challenges, and how to address them, will a) allow educators to develop more effective simulation-based training programs and b) provide researchers with a framework to design and conduct studies to establish clear linkages between simulation-based training (or assessment) and patient safety.
  
  Objectives: 
  1- To provide an overview of the use of simulation-based assessment in medicine 
  2- To outline some of the challenges associated with implementing simulation-based training and assessment
  3- To offer guidance on conducting research to show if, and how, simulation-based educational activities impact patient safety
   
• Workshop 4: Research in Simulation-Based Education
  In most health professions education programs, especially those that have embraced simulation in their curriculum, there is a growing need to understand pedagogical processes and how these relate to competence and patient outcomes.  As a result, educational research in the healthcare professions is rapidly advancing and is routinely conducted to solve practical problems and guide quality improvement activities.  Research topics in simulation-based education are quite diverse and include those related to skills acquisition and decay, curriculum design, assessment methods and psychometrics, and student selection, just to name a few.  While a host of research designs can be employed, including experimental and non-experimental, descriptive studies, where data are readily available or collected from surveys, case studies or personal observations, are most common.  In terms of analysis strategies, both qualitative and quantitative methods have been employed.  For qualitative methods, techniques include observations, in depth interviews and focus groups.  For quantitative methods, everything from basic descriptive statistics to more complex causal analyses has been employed.  Ultimately, the choice of research design, and associated analysis strategies, will depend on the purpose of the research and, more important, the specific questions that need to be answered.
  The purpose of this workshop is to provide the participants with a basic overview of research methods that can be applied to common questions concerning simulation-based education.  The target audience is individuals who are interested in research and want to know how to structure their activities and to avoid common pitfalls.

Learning objectives:
Following the workshop, participants will be able to:
1- Articulate meaningful research question
2- Choose appropriate research strategies
3- Understand, critique, and evaluate the design of research studies
4- Know the strengths and weaknesses of various research strategies
5- Know the publication outlets for simulation-based education studies

• Workshop C: Developing and Scoring Simulation-Based Assessments
  In medicine, and other allied health professions, various types of simulations have been used to educate and evaluate students and practitioners.  In many countries, these simulations, including those involving standardized patients, computer-based case management scenarios, and electromechanical mannequins, have also been employed as part of the certification and licensure of physicians.   While much work has been conducted to investigate the psychometric properties of the scores from these types of assessments, there remains some confusion regarding the choice of metrics for these types of performance-based assessments.  Moreover, given the vast array of simulation modalities available to educators, and the varying purposes of simulation-based assessment, guidelines concerning the proper construction of content-valid scenarios have not been well-articulated.
  Developing meaningful scoring rubrics for simulation-based assessments can be challenging.  Depending on the skill, or skills, being measured, analytic (e.g., checklists, key actions) and holistic tools (e.g., rating scales), or various combinations, can be employed.  The choice (or development) of scoring tools will depend on a number of factors including, amongst others, the purpose of the assessment, the specific proficiencies being evaluated, and the availability of qualified raters.  Regardless of the specific format of the evaluation tool, or the logistics of gathering the data, the scores, if used to make inferences concerning ability, must be reliable and valid.  The proper construction of scoring tools is a cornerstone of the validation process.    

  The workshop will consist of the following parts:
  1- Overview of simulation methods. The participants will learn about different simulation modalities and how, in general, to develop appropriate clinical scenarios.  Some examples from existing educational and assessment programs will be provided.
  2- Introduction to scoring and scaling issues. The participants will be provided with a brief overview of scoring and scaling processes, including the need to explicitly define what is to be measured, to delimit measurable skills/behaviors, and to choose appropriate metrics.  
  3- Developing a scoring tool.  As part of this workshop, the participants will develop a scoring tool for a simulation-based clinical skills scenario. 
  4- Scoring activities.  To better understand scoring issues (e.g., inter-rater reliability), and logistical issues associated with gathering performance data, participants will evaluate a series of clinical encounters.  
  
  Learning objectives:
  Following the workshop, participants will be able to:
  1- Choose appropriate simulation modalities for the competencies that they wish to evaluate
  2- Understand, critique, and evaluate the design of performance-based exercise
  3- Know the strengths and weaknesses of holistic (global) and analytic scoring systems and recognize where they can be best employed to gather valid and reliable assessment data
  4- Select appropriate scoring strategies and propose defensible data collection methods for typical simulation scenarios.

• Workshop 9: Teaching and Assessment Using Simulation
  Various simulation modalities, including standardized patients (SPs), mannequins, and part-task trainers are currently employed to educate and assess healthcare workers.   While even well-constructed simulations cannot replace interactions with “real” patients, they do provide a standardized milieu in which to assess specific skills or competencies, give and receive feedback, and gather data for program evaluation.   More important, one can create situations where trainees can practice techniques without the possibility of harming the patient.  
  Although simulations can be constructed to model many situation relevant to health professions education, there is relatively little guidance available to inform programs (e.g., medical schools, nursing schools) about their most effective and efficient use.   Also, while some countries have the resources to build and maintain sophisticated simulation centers, others do not.   Nevertheless, it may still be possible to develop simulation-based educational programs is resource-limited environments.                      
  The purpose of the workshop is to provide an overview of simulation methods that can be used to teach and assess healthcare workers.   Issues related to the costs and benefits of specific modalities, developing meaningful simulation content, logistics, the provision of meaningful feedback, and impact on patient care will be discussed.
  
  The workshop will consist of the following parts:
  1- Overview of simulation-based teaching and assessment
  2- Use of simulations for training and assessing healthcare workers
        a.  Discussion amongst participants on the use of simulations in their local environments (challenges and opportunities)
  3- Small group exercise to construct a simulated scenario that could be employed to train or assess healthcare workers
  
  Learning Outcomes:
  Upon completing the workshop, participants will:
  1- Understand the strengths and weaknesses of various simulation modalities
  2- Be able to determine the types of simulation tasks that could work in their local environment
  3- Have the basic skills needed to initiate the development of valid simulation-based training and assessment tasks
  

Katharine Boursicot, MD

Associate Professor and Assistant Dean for Assessment and Medical Education Research
Lee Kong Chian School of Medicine
Singapore, Republic of Singapore

Abstracts:

• Plenary 5: Performance assessment at undergraduate and postgraduate levels: the OSCE
  Since the OSCE was first introduced in 1975, it has become the most common way to access observed performance of clinical skills in a wide variety of health professional training and accreditation situations around the world. This plenary will chart the evolution of OSCEs and some key issues which have emerged over the last 40 years.
• Workshop 3: Teaching and assessing professionalism
  The aim of this workshop is to facilitate a better understanding of teaching and assessing professionalism in the undergraduate medical curriculum, although the general principles are applicable to the postgraduate area.  As part of the workshop, we will develop an understanding of the issues of professionalism in medical students. Building on this we will look at the elements of professionalism that are desirable and how to develop a program of assessment to ensure graduating doctors are aware of and committed to these values in their working lives. Finally will also examine the possibility of remediation of poor professional behavior.
• Workshop B: Tips for improving your OSCE stations
  In this workshop, we will offer some insights and suggestions drawn from 20 years’ experience of designing and implementing OSCEs at undergraduate and postgraduate levels. Topics such as length of stations, design, different scoring approaches, timing, circuit set-up and management as well as quality assurance issues will be covered.
   

Janet Grant, PhD

Honorary Professor, University College London Medical School
Professor Emerita of Education in Medicine, The Open University, UK
Director, Centre for Medical Education in Context [CenMEDIC] & FAIMER Centre for Distance Learning

Abstracts:

• Plenary 2: Medical education and the healthcare service: A lifelong symbiosis
  The quality of health care can only be judged by its practice. Medical education will be effective to the extent that it focuses on healthcare delivery. This is fundamental to the idea of social accountability. 
  The influence of medical education on the ultimate quality of health care is limited:
  • The predictive power of pre-admission variables for selection to medical school is weak
  • Prior academic attainment is still the best predictor of future performance
  • No curriculum type shows better overall predictive value than any other
  • Every variable has decreasing predictive power over time

Becoming a doctor is a process of navigating a complex and changing array of knowledge, skills, performances and identities. It is not surprising that the educational activities at one stage have little predictive power beyond the end of that stage. So we need to think differently about the relationship between medical education and the healthcare service:

• Medical education is an induction into professional practice
•  Medical education takes place in the context of the health care service
•  The stages of medical education from basic to continuing professional development (CPD)  must all be taken into account
• At every stage, the context of the healthcare service must be paramount: in terms of ensuring a contextual curriculum, to managing increasing independence at postgraduate level and education arising from the requirements of practice at CPD level
• The quality of health care is primarily a function of the service

The contextual nature of medical education means that we must focus equally on the educational process and on the health care service itself because being a good doctor is learned in the context of the service, and from the practice and role models observed there.

• Workshop D: Managing Change in a Medical Context
  Effective change management is highly contextual. In planning how best to manage change in a medical environment, we must take into account:
  •  The role and skills of the change leader
  • The characteristics and challenges of the professional context
  • The nature of the change intended

This workshop draws on published research on managing change specifically in a medical environment. The workshop with guide participants to consider their own contexts and skills and to learn how to use some change management tools such as:

•  network analysis
•  people charts
•  force-field analysis

We will consider:

•  the special challenges for change managers in a medical context
• the nature of academic leadership and the characteristics of medical professionals
• developing a vision for change
• the power that you have as a change leader
• orders and types of change
•  frequently cited factors in successful change management
•  core activities and tactical choices for the change manager

This highly participative workshop is based on research which developed the 10-step model of change management in medicine.
Reference: Gale, R. and Grant, J. (1997) AMEE Medical Education Guide No. 10: Managing change in a medical context: Guidelines for action. Medical Teacher 19, 4, 239-249.

• Workshop 5 : Continuing Professional Development for Health Service Improvement
  Continuing professional development [sometimes called continuing medical education] is a tropic of increasing importance. Doctors in practice have both a professional and a practical duty to keep up to date and to fill in any gaps in their learning. In many countries this is now a condition for having a continuing licence to practice.
  This workshop will look at:
  •  The conditions for CPD in participants’ own contexts
  • The purposes of CPD
  • The characteristics of effective CPD
  •  How to identify learning needs, to plan and undertake learning and then take that learning back to the workplace
  •  How best to manage CPD for the doctor, the health care system and the regulator
  • How to ensure that CPD delivers benefits for patient and the healthcare service.

The workshop will be interactive, and will be based on evidence from the literature and from professional practice.
Reference: Grant, J. (2012) The Good CPD Guide. 2^nd edition. Radcliffe Publishing, London.

•  Workshop 8: Developing Diagnostic Skills
  This interactive workshop will enable participants to understand the process of clinical problem solving from the point of view of a learner and an expert clinician. The aim of the workshop is to develop skill sin helping students and postgraduates to improve in this area.
   We will consider examples of normal, effective clinical problem solving as well as looking at common errors and considering why these occur.

The workshop will include a variety of exercises that participants will be able to use with their own students and trainees.  In addition, participants will be invited to complete the online Diagnostic Thinking Inventory after the meeting, and to have their students and trainees also complete it. This instrument enables people to measure the structure and flexibility of their thinking in relation to making a diagnosis.

The workshop will highlight the importance of the student’s and doctor’s memory, and how to improve access to the knowledge stored there within the clinical interview.

Typical scenarios will be offered that illustrate the challenges of simultaneously conducting a clinical interview, communicating with the patient and making a diagnosis.

The workshop will be practical, but is based on a strong theoretical framework.

John J. Norcini, PhD

President & CEO, Foundation for Advancement of International Medical Education & Research (FAIMER)
Philadelphia, USA

Abstracts:

• Plenary 4: National assessment programs and quality of care
  Many countries are exploring the use of national assessment programs for the licensure and certification of individual health care practitioners. In countries that already have such programs, there has been a movement to time limit these credentials so that practitioners must be reassessed periodically. National assessment programs are often time-consuming and unpopular with the participants. However, they are justifiable if they make a difference in the quality of patient care. This talk will explore some of the research around this question. 
   
• Workshop A: Teacher Assessment: Methods for the educational workplace
  The quality of health professions education is an issue of ongoing concern in many countries and central to it are the teaching skills of faculty. Methods for assessing and improving such skills are in their infancy, but many of those in use are the same as, or variations on, workplace-based assessments that are used with students, trainees, and practicing doctors. This workshop will survey these methods and discuss a variety of issues in their deployment. Active involvement will be encouraged throughout and small group exercises will focus on making judgments about actual teaching portfolios.
   
• Workshop 2: How to build an examination: the basics
  The quality of a test depends on how well it has been developed. The goal of this workshop is to provide beginners a systematic, eight-step process for test construction. It will touch on topics such as test purpose, test content, item format, testing time, and administration. In addition, it will describe the seven factors influencing good assessment. Both small and large group exercises will be conducted throughout and active participation will be encouraged.
   
• ​Workshop 6: A beginner’s guide to setting standards
  The goal of this session is to familiarize the participants with the major methods for setting standards [i.e., selecting the pass-fail point(s)] on written and clinical examinations. The session will start with a very brief overview of standards, how they differ from scores, the two types of standards, and the characteristics of a credible standard. The second part of the session will focus on specific methods of setting standards, including Angoff’s method and the contrasting group method. Steps in the implementation of each will be described. Active engagement will be encouraged throughout. 

Trudie E. Roberts, MD

Professor and Director, Leeds Institute of Medical Education
President, Association for Medical Education in Europe (AMEE)
Leeds, UK

Abstract:

• Plenary 7: Professional integrity: challenges and consequences
  Cheating is commonly defined as breaking the rules to gain advantage. How common is cheating in medical school examinations? It probably occurs more frequently than we would like to think. Why do students do it and how do they justify it when found out? Is cheating more morally wrong in would-be doctors than in other students? Are some types of cheating worse than others? How can we expose this type of deception and how can we deter students from deciding to cheat in assessments. How blameless are Faculty when students cheat? Indeed do they sometimes collude in cheating and why are they so reluctant to report cheating when it is discovered. In this presentation I will explore these issues and look at the dilemma posed by medical students, trainee doctors and Faculty who are academically and professionally dishonest. 
   

Ara Tekian, PhD, MHPE

Associate Professor and Director of International Program (DME)
Associate Dean, Office of International Education
University of Illinois at Chicago, College of Medicine

Abstracts:

• Plenary 3: Creating Leaders and Scholars to Improve the Quality of Healthcare
  The challenge of improving the quality of healthcare is steadily getting more complex and expensive.  Institutions and organizations have tried to deal with this issue in many different ways, such as introducing innovations in the continuum of medical education, improving the quality of teaching and assessment at both undergraduate and postgraduate levels, and monitoring the competence of practicing physicians through observations and requirements for maintenance of certification. 
  Currently, North American countries are facing a number of challenges and there are lessons to be learned.  The Institute of Medicine’s recent report (Graduate Medical Education that Meets the Nation’s Health Needs) released in September 2014 emphasizes the allocation of resources for graduate medical education; the Accreditation Council for Graduate Medical Education (ACGME) monitors the competence of graduates through assessment of specialty specific milestones; and the Association of American Medical Colleges published a list of 13 Entrustable Professional Activities (EPAs) to ensure the competence of graduates from medical school.    Additionally, the Educational Commission for Foreign Medical Graduates (ECFMG) finalized their deadline that as of 2023 no medical graduate will get ECFMG certification if they are not graduating from an accredited medical school. 
  In light of these challenges, formal training programs in health professions education have been growing exponentially worldwide.  For example, during the past two decades because of increased demand, there has been a proliferation of Masters’ and doctoral programs in health professions education.  This demand has been due to an increased need for leaders who have knowledge and competence in the theory and practice of medical education, along with an increased need to have teaching faculty who are up-to-date with the literature in HPE, have acquired the necessary skills to be effective teachers, and could act as resource individuals in their institutions.  Additionally, recent developments and requirements by accrediting bodies - ACGME and the LCME - for institutions to meet criteria for excellence in curriculum, instruction, assessment and evaluation have increased the demand for leaders with applied knowledge in these areas in most institutions, hospitals, associations, and agencies. 
  A recent longitudinal study capturing the performance of medical students and graduates from Arab countries confirmed that the number of Arab trained students/graduates seeking ECFMG certification is increasing.  This means that the quality of medical education and training programs needs special attention to ensure success of these students and residents that in turn improve the quality of healthcare.  Preparing leaders and scholars is one way of handling this challenge, which will be the focus of this presentation.   
   
• Workshop 1: Sources of Error during Verbal/non-Verbal and/or Written communication
  Communication is a critical component of the health care system and a fundamental underpinning to patient safety. Communication, traditionally defined as a simple exchange of information, or a “message” between a “sender” and a “receiver,” is now considered a complex process with verbal and nonverbal components, requiring interpretation and inferences of meaning, synthesis of information, and translation into situational context. In our current culture, communication takes on many forms—including oral and written—and is often presented by intermediaries (think of those hundred or so e-mail exchanges each day).  In recent years, attention has focused on the frequent occurrence of clinical errors in hospital- and office-based medical practice. Although it was common practice in the past to cover up such mistakes, today it is widely accepted that patients should be informed when errors occur. The Joint Commission for Accreditation of Healthcare Organizations makes this an explicit requirement. Communication—whether verbal, nonverbal, written, and via technology (phone messages, e-mails, etc.)—is a core component of both medical error and quality improvement.  During this workshop we will look more closely at how each of these elements plays a role in the evolving story of medical error.   The goal of this workshop is to understand the importance of communication between doctors and patients and how poor communication can be a source of error.  You will watch some videos and identify sources of error, and discuss ways of disclosing errors.
• Workshop E: Patient safety and medical errors 
  In 1999, the Institute of Medicine (IOM) report estimated as many as 98,000 patients die every year from preventable medical errors in U.S. hospitals. Medical errors are defined as the failure of a planned action, or the use of the wrong plan, to achieve an aim. To ensure patient safety, health care providers and systems must be prepared to recognize potential sources of error, to acknowledge vulnerability to error, and to fully engage in the process of continuous quality improvement.   Most providers agree that medical errors should be disclosed to patients; however, research demonstrates that disclosure of errors is uncommon, with roughly only 1 in 4 errors being disclosed. Many providers continue to remain silent secondary to the fear of litigation, fear of stating explicitly to a patient that an error occurred, and the desire to put a positive spin on a situation. Addressing quality and safety issues in health care is difficult because the culture of secrecy and silence reduces awareness by future generations of providers.
  This workshop will explore in depth error science, and elaborate issues related to reporting, investigation, Root Cause Analysis, and quality improvement.  By observing various segments of an unfolding real story, the participants will be able to identify a number of errors and explore ways of avoiding them.  This workshop will be very interactive, and at the conclusion participants will be asked to reflect on ways incorporating the discussion of different types of error into the undergraduate and postgraduate curriculum. 
• Workshop 7: Getting published
  The purpose of this interactive session is to provide a practical understanding of where and how to publish and present scholarship in health professions education.  The first part of this workshop will review the research trajectory of five educators who started their career by identifying one research questions, which eventually led to creating an entire research agenda.  The second part will focus more on understanding criteria against which publications and presentations are judged, and will identify venues for scholarship in health professions education.  Participants will review each other’s research hypothesis and answer the questions “So what? Who cares?”