Asian Journal of the Scholarship of Teaching and Learning

The Asian Journal of the Scholarship of Teaching and Learning (AJSoTL) is an international, peer-reviewed, open-access, online journal. AJSoTL seeks to create and nurture a global network of academics and educators who will discuss ongoing changes and future trends in tertiary education.

The Evaluation of Nursing Competency in a Simulation-based Assessment: Tool Development and Students’ Experiences

Articles, Original Article

TAN Khoon Kiat, Sabrina PALHAM, Jeanette IGNACIO, Rabiah Binte Mohamed Dawood, Sandra MACKEY, LIM Fui Ping, LIAW Sok Ying


TAN Khoon Kiat - School of Health Sciences (Nursing), Nanyang Polytechnic, Singapore


Sabrina PALHAM - Metro South Health, Queensland, Australia

Jeanette IGNACIO, LIM Fui Ping, LIAW Sok Ying - Alice Lee Centre for Nursing Studies, National University of Singapore

Rabiah Binte Mohamed Dawood - School of Sports, Health and Leisure, Republic Polytechnic, Singapore

Sandra MACKEY- School of Nursing and Midwifery, University of Western Sydney, Australia


Address for Correspondence:
Dr Liaw Sok Ying RN, MHScED, PhD, Alice Lee Centre for Nursing Studies, National University of Singapore, Level 2, Clinical Research Centre, Block MD11, 10 Medical Drive, Singapore 117597

Email: nurliaw@nus.edu.sg

 

Recommended citation:

Tan K. K., Palham, S., Ignacio, J., Dawood, R. B., Mackey, S., Lim, F. P., & Liaw, S. Y. (2016). The evaluation of nursing competency in a simulation-based assessment: Tool development and students' experience. Asian Journal of the Scholarship of Teaching and Learning, 6(2), 221-245.

ABSTRACT

Background: Prior to this study, skills assessments had often been task-based and technical skills assessed using checklists in isolation from other competencies associated with holistic nursing. Aim: The aims of this study are twofold: to develop and determine the psychometric properties of a simulation-based assessment tool, and to explore students’ experiences with the use of simulated patients in a simulation-based assessment. Method: A simulation-based assessment tool was developed to examine six core competencies expected of registered nurses—namely, critical thinking, communication, technical skills, management of care, safe practice, and professionalism and ethical practice. Scenario-based learning and assessment sessions with simulated patients were introduced within the context of assessment and to build mastery of the core competencies. The cohort, comprising 89 Year One nursing students, participated in the evaluation of the simulation-based assessment approach. The survey questionnaires administered used the Maastricht Assessment of Simulated Patients and the Objective Structured Clinical Examination Evaluation Questionnaire. Conclusion and implications: Despite the stressful nature of the new assessment method, it was well received by the students, who reported that it enhanced their learning and helped them develop the necessary competencies required of the degree nursing programme. The comprehensive simulation-based assessment tool and the innovative assessment methodology addressed the erstwhile unassessed competencies required of a registered nurse. The approach has since been introduced into other nursing skills modules across the nursing curriculum.

INTRODUCTION


Student nurses are often required to perform skills in learning and assessment 
situations as part of their preparation for professional practice. Traditionally, these processes have focused on isolated technical skills, e.g., performing a wound dressing on a mannequin. Alternatively, skills were performed on peers who role-played with little preparation and with variable quality, achieving indeterminable learning outcomes. Actual clinical encounters with patients, on the other hand, demand integration of knowledge, technical and communication skills and the demonstration of professionalism. To be more reflective of real clinical practice, there is a need for learning and assessment to examine competence across multiple domains. The innovation of simulation methodology provides the prospect of incorporating a more complex clinical situation that allows opportunities for learning and assessment of cognitive, psychomotor and affective competencies expected of a nurse in a more realistic way (Liaw, 2011).

The utilisation of Simulated Patients (SPs), standardised and professionally prepared to portray real cases, or the hybrid employment of SP with task trainers (e.g., an open wound), allows for the learner’s interaction with the patient while performing a skill (Flynn, 2012). The rigorous training of the SPs is important to achieve the accuracy and consistency needed for realism (Erby, Roter, & Biesecker, 2011). With the use of SPs, the focus essentially shifts from sole concentration on a technical skill to caring for a person with a condition or with needs to be met. Consequently, this mimics the competencies essential in holistic caring within clinical settings. It is believed that the use of an SP can enhance learner satisfaction and ensure greater objectivity of the education process (Ebbert & Conners, 2004).

According to Norman (1985), competency refers to an expected level of performance that results from the integration of knowledge, technical skills, communication skills, and problem solving abilities. The competency-based approach to education has been evident in nursing. Mastery learning is considered an important approach to this competency-based education (Mcgaghie, Issenberg, & Petrusa, 2010). It is believed that educators who adopt this philosophy—along with group-paced instruction involving the appropriate selection of content, teaching and evaluation of learners’ progress—would achieve equal measures of publicly defined and criterion-based educational outcomes. A meta-analysis supports the effectiveness of mastery learning in summative educational outcomes (Kulik, Kulik, & Bangert-Drowns, 1990). However, development of true competence requires both mastery of learning philosophy and pedagogy to enhance knowledge, and proficient application of knowledge in the mastery of skills through learning and assessment (Ebersole, 2014). These must be closely scrutinised for undergraduate nursing programmes.

As a result, a new tool was developed to assess clinical units based on the incorporation of core competencies and the engagement of SPs. There is also a need to evaluate students’ perceived benefit of this approach.

AIMS


The aims of this study are twofold: to develop and determine the psychometric properties of a simulation-based assessment tool (SAT), and to explore students’ experiences with the use of SPs in a simulation-based assessment.

METHODS


The first phase of this project involved the development and validation of the SAT. Next, an innovative scenario-based learning and assessment methodology was designed to foster and appraise a comprehensive range of core nursing competencies using the SAT. The simulation-based assessment approach was then evaluated for its acceptability and perceived benefits.

Development and validation of the SAT


Based on the Singapore Nursing Board (SNB) guidelines (Singapore Nursing Board, 2012), nursing education literature on core competencies of registered nurses, and an established clinical assessment tool known as the Mini-CEX (American Board of Internal Medicine, 2014), the team identified 
six domains of performance: critical thinking, communication, technical skills, management of care, safe practice, and professionalism and ethical practice. Guided by specific performance criteria, a nurse is expected to demonstrate these competencies during the delivery of nursing care. The six domains provided a framework for the development of the SAT.

A checklist of 33 items was generated for these six domains in the first draft of the SAT. Each item has a 3-point scale: 0 = not performed; 1 = performed but not competent; 2 = performed competently. These items were condensed to an individual global rating scale for each domain. The global rating scale consists of a 9-point horizontal line, segmented by 3 descriptors: unsatisfactory (1 to 3); satisfactory (4 to 6); outstanding (7 to 9).

Subsequently, the SAT was reviewed by a panel of ten nurse educators from four educational institutions in Singapore. They rated each item on a four-point scale: 1 = not relevant; 2 = item needs some revision; 3 = relevant but needs minor revision; 4 = very relevant. They also gave their comments for the items and domains, and suggested additional items for inclusion. The item-level content validity index was 0.8 and above as assessed by the experts. The content validity index for the 35 items was 0.97. Revisions were made to several items based on the experts’ comments.

Content and face validity evaluation of the SAT was undertaken by a team of nurse educators. After a briefing on the assessment criteria of the SAT, they independently rated video recordings of two clinical encounters: one designed to elicit a good rating, and the other, a poorer rating. At a joint discussion, the evaluators unanimously reported difficulties in accounting for the individual criterion when using the lengthy checklist. Conversely, they found the global rating scales relatively easy to score, where the entire performance was rated based on six core competencies. However, when using both the checklist and global rating scales, focusing on the individual checklist items often diverted them from their expert judgement in rating the overall performance. After much discussion over the two scoring systems, they decided to concentrate solely on the global rating scales for measuring all the constructs. The items from the checklists were converted to descriptors that specifically defined the global rating scales (See Appendix).

To establish inter-rater reliability of this SAT, two assessors were assigned to rate the performance of each student during a clinical skill performance assessment. These assessors were academic staff from the nursing profession and were involved in teaching Year One nursing students. They were prepared in advance as they were new to this assessment method. The assessors were first briefed on the rationale for the change in assessment method and the development of the SAT. Next, they were shown two videos recorded during the mock assessment—one good and one poor performance—and were required to familiarise themselves with the use of SAT by evaluating students’ performance. Discrepancies in scoring were discussed to establish consensus on the expectation of students’ performance, thus ensuring consistency among the assessors in the use of the SAT.

Table 1
Intra-class correlation of six competency domains

 

The tool was found to have very high inter-rater reliability. Intra-class correlation (ICC) ranged from 0.719 to 0.929 for the six competency domains (Table 1). Among all the competencies, “technical skills” had the highest ICC of 0.929, indicating a good correlation. “Professionalism and ethical practice” had the lowest ICC of 0.799, perhaps suggesting the relative rater-subjectivity of this domain and inadequate assessor/rater preparation. The overall ICC was 0.900.

Implementation of the SAT with simulated patients


The nursing students who participated in this study attended the foundation of nursing modules in their first year of the Bachelor of Science (Nursing) programme. One of the modules in this programme was the skills-based module where students were required to integrate their learning and demonstrate their competence during the clinical skill performance assessment. The SAT was introduced early in the module as a learning tool to emphasise the importance of comprehensive competence while performing nursing care. Students were required to demonstrate beginner competence in the six domains of professional practice.

The module comprised various learning methods such as lectures, problem-based learning tutorials in which students receive guidance in building their research and reflection skills, and laboratory practice with case scenarios, peer role-plays and simulated patients. Based on the philosophy of mastery, skills acquisition was developed through scaffolding activities and deliberate practice (Archer & Hughes, 2011). After a demonstration of the technical skills during laboratory practices, learners first practised these isolated skills before being grouped in threes to practise with one student playing a scripted patient; the second, a peer assessor using the SAT; and the third, a nurse caring for the patient. A drawback during these laboratory sessions was that student nurses might not be serious with their practice because of familiarity with their classmates who role-played as patients. Subsequently, a mock assessment was introduced with student SPs from a senior cohort, enabling student nurses the opportunity to undergo simulated practice with SPs. Thus, they became more familiar and less apprehensive with simulation-based assessment using SAT as an assessment tool. During the actual assessment, the scenarios used were different from the mock assessment. Student SPs that participated in both the mock and actual assessment had to go through an SP training session conducted by a trained SP coordinator to prepare them for their roles. During training, the fidelity of the SPs was validated through exposure to different encounters.

Evaluation of the simulation-based assessment

Design and sample

A descriptive cross-sectional design study was adopted. Ethical approval was obtained from the University’s Institutional Review Board. Eighty-nine student nurses in the Year One cohort who had obtained similar levels of nursing training were informed in advance of the purpose of the study via email by administrative staff. Immediately after the clinical skill performance assessment, a research assistant (RA) ushered students who agreed to participate in the study into a quiet room. The RA distributed a copy of the participant information sheet to every prospective participant and informed them that their participation in the study was voluntary. Verbal informed consent was sought. All 89 student nurses returned the completed questionnaires. There were no missing values.

Instrument and data collection

In addition to demographic data, two other instruments were included in the data collection process. The Maastricht Assessment of Simulated Patients (MaSP) tool was developed and validated with a Cronbach’s alpha of 0.73 by Wind and colleagues in 2004. This tool consists of two sections: authenticity during the consultation as well as feedback after the consultation, the first of which was adopted with the exception of the last item (“SP starts conversation with the student(s) during time-out”) as it was deemed irrelevant. This was followed by a free-text section with the heading “Comments on SP”. The second tool to be adopted was the Objective Structured Clinical Examination Evaluation Questionnaire by Pierre and colleagues (2004), incorporating the rating scale as modified by Selim et al (2011). Again, this was followed by an open-text section with the heading “Comments on your experience”, designed to collect descriptive input to enrich the data. Permission was sought to further modify the stems in order to accommodate local semantics, i.e., “exam” was changed to “assessment”. At the end of the questionnaire, the students marked a visual analogue scale indicating the worthiness of the simulation-based assessment experience.

FINDINGS


The findings include a brief demographic background of the participants, the perceived authenticity of the portrayal of the simulated patients, and the students’ evaluation of the simulation-based assessment experience.

Demographics


The participants were aged between 18 and 28 years with a mean of 20 (SD = 1.496); about 75% (n = 67) were women.

Authenticity of simulated patients


Table 2

Students’ evaluation of the simulation-based assessment 


The students’ perception of the authenticity of SPs was found to be very positive (Table 2). A vast majority agreed that the SPs appeared authentic (95.5%, n = 85), stayed in their roles (96.6%, n = 86), and were natural in their presentation (91%, n = 81) and the manner in which they answered questions (94.4%, n = 84). About 30% of the students reported that the SPs did not portray the physical complaints realistically (29.2%, n = 26), and a quarter perceived that the SPs were withholding information unnecessarily (25.8%, n = 23). Despite some reservations regarding the authenticity of the SPs, most students agreed that the SPs might have been real patients (79.8%, n = 71).

Descriptive data on the authenticity of the SP were collected and thematically analysed by two researchers. Three themes emerged, and they are: “acting real”, “cooperating”, and “learning with SP”.

In the theme “acting real”, despite students’ awareness of the fact that the SPs were actors, they shared a willingness to suspend reality and were able to discern the quality of the SPs’ performance and appreciated the need to respond appropriately. One stated:

“Very authentic. The way she presents the problem (sore-throat) makes me feel that this is a real patient and that I need to think fast in order to help my patient.”

The students also noticed the emotional component of the SPs’ performance and its contribution to the overall portrayal, and how this influenced their approach. One student stated:

“The SP could possibly be more demanding and not give in. For example, accidentally[sic], I spilled the feeding on the SP and as a patient, naturally they would get slightly angry, but the SP was relaxed.”

The theme “cooperating” highlights the inevitable interplay between the SP and the student in their roles as the assessment tool and the candidate, respectively. Many students judged the cooperativeness of the SPs during the assessment and in some instances, how this might have aided their performance. As one student stated:

“The SP was good at role-playing, and I can feel that she was not trying to make the case difficult for me, and I am thankful for that.”

Another student captured this by stating:

“She knew I was struggling, so she helped me a bit which was good as I felt very stressed.”

The students also expressed how an authentic rendering by the SPs during the summative assessment created a positive environment that enhanced their “learning with SP”, which could potentiate their future nursing practice. As one student elaborated:

“SP did a good job on non-verbal cues that helped me to critically think and act on the spot. Though I may not be very proficient in my skills, this to me is considered one good clinical experience that I will remember.”

In another student’s words:

“SP played a role that is quite common in most problematic patients. Would be better to equip myself with the skills to deal with the real life patients."

Students’ evaluation of the simulation-based assessment experience


From the visual analogue scale, the overall experience of the simulation-based assessment was worthwhile; with a mean score of 6.78 (SD = 2.099, n = 89). This demonstrated that the students found the method of assessment using simulation to be valuable to their learning.


Table 3
Assessment of SP authenticity by students

The findings of the students’ evaluation of the simulation-based assessment are presented in Table 3. In general, the students agreed that the assessment was satisfactorily structured (100%, n = 89) and administered (97.8%, n = 87). About 60% of the students were satisfied with the clarity of instructions. They were fully aware of the level of information needed and the nature of the assessment. Ambiguously, 61.8% (n = 55) indicated that the time spent at each station was adequate, yet most would have preferred more time (81.9%, n = 73).

Majority of the students appraised the assessment as fair (64%, n = 57), and that the tasks assessed were appropriate and reflective of the scope of the curriculum (70.8%, n = 63), and pitched at a level befitting a novice (74.2%, n = 66). The students judged the scenario used in the assessment as logical (80.9%, n = 72), although the physical setting of the assessment was not seen to be very authentic (56.2%, n = 50).

The simulation-based assessment was a stress-inducing experience. It was experienced by two-thirds of the students as being very stressful and more stressful than other assessment methods. Only 13.5% (n = 12) indicated that the assessment was not at all intimidating.

According to the majority of the students, the assessment examined an adequate scope of knowledge and competencies (94.4%, n = 84). Despite the summative nature of the assessment, most of the students acknowledged that it provided a platform for learning (85.4%, n = 76) and highlighted some areas requiring further improvement (71.9%, n = 64).

The descriptive data on the simulation-based assessment experience revealed three themes: “internalisation of learning”, “recognising the influence of emotions”, and “assessment environment”.

In the theme “internalisation of learning”, the students recognised a transition towards becoming a competent practitioner, requiring a deeper understanding of their individual learning process. The students stated their desire to take individual responsibility for their learning; they expressed that the assessment enabled them to understand their current level of competency, and initiated within them a drive to improve. One student stated:

“My experience has been rather enlightening as it shows that I am still inadequate in some of the skills and lack critical thinking when performing.”

In another student’s words:

“Although it was stressful and challenging due to [it being my] first experience, it encourages me to want to improve on my clinical and practical skills.”

Some students realised that the range and complexity of skills can only be acquired through deliberate practice and holistic assessment with SPs. One elaborated:

“Overall, I think that this makes me realise that knowing the procedure is not enough, we need to practise more often. This is because you might think that you know what to do, but [when it] comes to real life situations, technique is very important. Therefore, I believe that it is good that we have simulation-based assessment test.”

Apart from technical skills, the students indicated the importance of other skills such as situation awareness, critical thinking and communication. There was an added realisation of the need to integrate skills for the development of a therapeutic relationship and satisfactory management of care. One student captured this by stating:

“I have realised that it is much more stressful to work with a real patient and the circumstance I am in is different and unexpected every day. Thus, it is not enough to just remember the procedure of the skills step by step. Indeed, critical thinking and quick reaction and communication are also very[sic] imperative. I think I will only be able to deliver care effectively and efficiently to my patient if I am able to master and incorporate all these mentioned skills.”

The students also recognised the value of the intensity of the assessment approach in the acquisition of the competencies expected of a practising professional. They suggested more progressive staging to help with the learning process. A student stated:

“It was definitely a learning experience for me. Performing and communicating under pressure is challenging. I will need more practice on that. I appreciate the school’s effort in building this into the curriculum.”

In the construction of an “assessment environment”, factors such as time, place and the people have inf luenced the outcome. Many students recognised the potent learning environment inherent in the assessment context. The authenticity of the assessment environment posed a unique challenge and influenced performance. One student elaborated:

“The environment was largely realistic, and it really test(s) my observation skills and sensing of objects around me even though I am focused with my task at hand. There are many considerations at any one point [in] time, which really challenges my multi-tasking skill. Though I may tend to think of it as an assessment, I was trying very hard to imagine this as a real clinical situation and make my task at hand patient-focused.”

Many students also recognised the value of a clinically accurate physical domain to promote a serious atmosphere for enhanced learning, while unrealistic elements distracted them from performing as well as they had hoped. One student stated:

“[The simulation has] given us a hands-on experience of how a real patient [would] react or behave. Some portions are not as well simulated like, for example, materials at the table are all related to a specific task, while in real life, the store room [would] contain all sorts of materials needed for all sorts of tasks.”

Having an assessor within the designated performance area was reported to have had an impact on the students’ learning experience. The assessors’ presence was perceived as a source of stress for some students, while others indicated that the demeanour of the assessor influenced their emotional state. While some assessors were reported to be kind or friendly, the facial expressions of some were perceived as unnerving. Some students gained from the assessors’ prompting and feedback, perceiving them as beneficial to their learning. For example, one student reported:

“Very good learning experience. Feedback from assessor helped me to realise where I was falling short; reminder of mistakes and what should be improved.”

The students also raised the issue of assessment time as a determining factor in their performance. Some shared the need for more time to practise, to immerse themselves in the scenario, and to deliver an acceptable level of competency within the assessment time frame. As one student stated:

“Not enough time to really practise my communication and technical skills. Not really used to it, so I think I need more time as this is my first time. It takes me a longer time to orientate myself.”

In the theme “recognising the influence of emotion”, the spectrum of emotions and their effects on students’ performance ranged widely. Some were overwhelmed, resulting in poor performance and self-assessment, while others acknowledged that the realism intensified the learning potential of the encounter whether or not they transcended the negative influence. One student reported thus:

“I was super nervous and I forgot to identify the patient!!! I don’t really know how to comfort the patient when she said she [was] worried about the colonoscopy.”

 In another student’s words:

“The experience was nerve-wrecking[sic], but certainly worthwhile, because it is an eye opener for me, in terms of seeing how I would react in real clinical settings and at the same time, it allowed me to spot my mistakes so I can learn from them and avoid.”

DISCUSSION


The introduction of simulation-based assessment in assessing nursing competencies has demanded the construction of a valid and reliable tool. Although simulation-based assessment enables the global evaluation of learning (including affective, cognitive and psychomotor learning domains), a lack of a valid and reliable simulation evaluation tool in nursing education has inhibited the use of simulation-based assessments (Kardong-Edgren, Adamson, & Fitzgerald, 2010).

In this study, a systematic and comprehensive methodology involving the use of the SNB guideline, peer-reviewed literature, expert consensus and psychometric testing was undertaken to develop and validate the SAT tool. Using the core competencies of nursing, six domains were identified for the SAT. Although the initial checklist items generated for these domains yielded an overall content validity index (CVI) of 0.97, the pilot testing identified the difficulties encountered by the evaluators in using the lengthy checklist. A consensus was therefore made among the nursing experts to rate the six core competencies using global rating scales and the checklist items converted into descriptors for defining the global rating scales. Rather than directing the evaluator’s attention to individual tasks, the global ratings allow evaluators to evaluate the performances as a whole involving expert judgement. Despite concerns regarding the reliability of a global rating scale (Boulet and Murray, 2010), our study demonstrated an excellent inter-rater reliability between the two evaluators. Before utilising the SAT tool, the raters must have a good understanding of the application of each core competency in the tested scenarios. Regardless of their clinical experiences and expertise, the evaluators needed to receive proper training or briefing on the use of the SAT to ensure their ratings accurately reflected the students’ performance.

In addition to simulation-based assessment, the SAT was also applied during simulation learning to guide faculty in providing holistic feedback to students. The SPs involved in the simulation-based assessment were carefully trained to present an illness or scenario. Majority of the student nurses agreed that the SPs were authentic, stayed in their roles, and were natural in their presentation.

These descriptions indicated that the student nurses perceived themselves to be managing real patients in an actual clinical setting. As such, their responses to the SPs may be very similar to how they would respond in real patient encounters (Flanagan & Joseph, 2004). SPs, as a teaching tool, are increasingly being used in healthcare education and with promising outcomes (Anderson, Holmes, LeFlore, Nelson, & Jenkins, 2010; Liaw, Scherpbier, Rethans, & Klainin-Yobas, 2012). It has been shown that SPs bring realism to simulation and this realism enables learners to perceive the simulation experience to be authentic (Ignacio et al., 2015).

Our study revealed that the student nurses’ knowledge of SPs’ play-acting did not dilute the authenticity of the SPs’ role play. In fact, because of this, the students felt the need to respond appropriately. While the qualitative data highlighted the influences of the emotional component of the SPs’ portrayal on the responses of the students, the quantitative findings suggest the possibility of eliciting realistic responses from the students toward the SPs. The added realism from the emotional component enabled the participants to have an enriching learning experience that would prepare them for actual patient encounters in the future (Ignacio et al., 2015; Jenkins & Schaivone, 2007).

In congruence to previous studies (Becker, Rose, Berg, Park, & Shatzer, 2006; Luctkar-Flude, Wilson-Keates, & Larocque, 2012), our study showed that the use of SPs in simulation inevitably produced anxiety and/or stress among the students. Interestingly, the students reported that even though they were under stress, the SPs did not add to their stress as these SPs did not make the cases more difficult for them. The SPs were meticulously trained and their responses were homogenised so that all students would have had the same experience. However, this seemed to facilitate the perception that they were “cooperating” with the students during assessment. Hence, the goal of utilising the SPs was not to add stress but to make the scenario engaging for the students and also as realistic as is deemed possible.

Staged learning, involving the use of scaffolding strategies to progressively integrate theory and skills, is more aligned to the mastery approach required in the development of practice. Our students recognise the need to achieve mastery of a wide range of complex skills to enhance their transition to a complex healthcare environment. The multiple skills required, including situation awareness, critical thinking and communication, need to be integrated for the fostering of a therapeutic relationship with their patients. In this context, the use of SPs during the simulation assessment has enabled our students to understand their current level of competencies through the internalisation of learning. The provision of deep insights, as well as a profound understanding of their own individual learning processes, has provided motivation for our students to direct their own learning. This learning experience portrayed by our students in this study is consistent with the meta-analysis study conducted by Oh and colleagues (2015), which identified the beneficial effects of SPs on the cognitive, affective and psychomotor domains of learning. Interestingly, the presence of an assessor was also reported to have an impact on the students’ learning experience, despite it being a source of stress for some. The students recognise the benefit of external feedback from multiple sources, and its influences in calibrating their self-assessment.

Moving forward from this study, more emphasis has been made on the simulation-based learning component of the curriculum to prepare our students for simulation-based assessment. This included helping our students to critically reflect on their simulation experience through faculty-led debriefings. Besides simulation-based assessment, the SAT tool was utilised by the faculty and peers during simulation learning to guide them in providing holistic feedback to the students. Finally, we now also prepare and involve the SPs in providing constructive feedback—specifically on interpersonal and communication skills—to our students, with the goal of enhancing their professional attributes.

Limitations


First, the psychometrics testing of the SAT tool was limited to content validity and inter-rater reliability. The ability of the SAT to differentiate the competency levels of nurses (construct validity) and to predict future performance (predictive validity) could be tested in future studies to strengthen the psychometrics properties of the tool. Second, data from the students’ perspectives were collected immediately after the assessment. This could be contentious because of the unknown influence participants’ emotions had on their responses to the questionnaire. Although the open-comment sections allowed students to qualify their quantitative responses freely, it was not structured to further the study aims. Finally, the students’ scores using the SAT were measured in a simulated environment which may not translate to real practice outcomes.


CONCLUSION


This study identified a simulation-based assessment tool that allows student nurses to demonstrate the integration of core competencies expected of a competent practitioner in the clinical arena. The assessment reliably and accurately appraises students’ performance in a veritable reality setting to allow assessors to discern nursing competencies from extraneous factors that may confound evaluation. The assessment tool is both acceptable and favourable to the learners. To better bridge the learning-practice gap, a review of how clinical units are taught in the first year of nursing programmes is indicated. The closer the learning environment is to the practice milieu, the more effective students’ learning will be.

 

Acknowledgements


This study was supported by Teaching Enhancement Grant from the National University of Singapore. The authors would like to thank the nursing students of NUS for their participation in this study. We also thank Jenny Yeo for her support for this study. Lastly, we would like to thank the National University of Health System, Publication Support Unit, Singapore, for providing editing services for this manuscript.


 

REFERENCES


American Board of Internal Medicine. (2014). Mini-CEX. Retrieved from http://www.abim.org/program-directors-administrators/assessment-tools/mini-cex.aspx.

Anderson, M., Holmes, T. L., LeFlore, J. L., Nelson, K. A., & Jenkins, T. (2010). Standardized patients in educating student nurses: One school’s experience. Clinical Simulation in Nursing, 6(2), e61-e66. http://dx.doi.org/10.1016/j.ecns.2009.08.001

Archer, A. L., & Hughes, C. L. (2011). Explicit Instruction: Effective and Efficient Teaching. New York: The Guilford Press.

Becker, K. L., Rose, L. E., Berg, J. B., Park, H., & Shatzer, J. H. (2006). The teaching effectiveness of standardized patients. The Journal of Nursing Education, 45(4), 103-111.

Boulet, J. R. & Murray, D. J. (2010). Simulation-based assessment in anesthesiology requirements for practical implementation. Anesthesiology, 112(4), 1041-1052. http://dx.doi.org/10.1097/ALN.0b013e3181cea265

Ebbert, D. W. & Conners, H. (2004). Standardized patient experiences: Evaluation of clinical performance and nurse practitioner student satisfaction. Nursing Education Perspectives, 25(1), 12.

Ebersole, J. F. (2014, July 25). Competency v.s. Mastery. Inside Higher Ed. Retrieved from https://www.insidehighered.com/views/2014/07/25/lets-differentiate-betweencompetency-and-mastery-higher-ed-essay.

Erby, L. A. H., Roter, D. L., & Biesecker, B. B. (2011). Examination of standardized patient performance: Accuracy and consistency of six standardized patients over time. Patient Education and Counseling, 85(2), 194-200. http://dx.doi.org.10.1016/j.pec.2010.10.005

Flanagan, B., Nestel, D., & Joseph, M. (2004). Making patient safety the focus: Crisis resource management in the undergraduate curriculum. Medical Education, 38(1), 56-66. http://dx.doi.org/10.1111/j.1365-2923.2004.01701.x

Flynn, K. (2012). The use of standardized patients to minimize anxiety in undergraduate nursing students in the clinical setting (Masters of Arts in Nursing Thesis , St Catherine University). Retrieved from
http://sophia.stkate.edu/cgi/viewcontent.cgi?article=1057&context=ma_nursing.

Ignacio, J., Dolmans, D., Scherpbier, A., Rethans, J., Chan, S., & Liaw, S. Y. (2015). Comparison of standardized patients with high-fidelity simulators for managing stress and improving performance in clinical deterioration: A mixed methods study. Nurse Education Today, 35(12), 1161-1168. http://dx.doi.org.10.1016/j.nedt.2015.05.009

Jenkins, L. S., & Schaivone, K. (2007). Standardized patients in nursing education. In M. H. Oermann, & K. T. Heinrich (Eds), Annual Review of Nursing Education: Challenges and New Directions in Nursing Education (pp. 3-23). New York: Springer.

Kardong-Edgren, S., Adamson, K. A., & Fitzgerald, C. (2010). A review of currently published evaluation instruments for human patient simulation. Clinical Simulation in Nursing, 6(1), e25-e35. http://dx.doi.org/10.1016/j.ecns.2009.08.004 

Kulik, C. C., Kulik, J. A., & Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs: A meta-analysis. Review of Educational Research, 60(2), 265-299. http://dx.doi.org/10.2307/1170612

Liaw, S. Y. (2011). The integration of simulation into undergraduate nursing curriculum. Technology in Higher Education: The State of the Art. Retrieved from  http://www.cdtl.nus.edu.sg/tech-in-he/pdf/Section3-Article1.pdf.

Liaw, S. Y., Scherpbier, A., Rethans, J. J., & Klainin-Yobas, P. (2012). Assessment for simulation learning outcomes: A comparison of knowledge and self-reported confidence with observed clinical performance. Nurse Education Today, 32(6), e35-e39. http://dx.doi.org/10.1016/j.nedt.2011.10.006

Luctkar-Flude, M., Wilson-Keates, B., & Larocque, M. (2012). Evaluating high fidelity human simulators and standardized patients in an undergraduate nursing health assessment course. Nurse Education Today, 32(4), 448-452. http://dx.doi.org/10.1016/j.nedt.2011.04.011

Mcgaghie, W. C., Issenberg, B., & Petrusa, E. (2010). A critical review of simulation-based medical education: 2003-2009. Medical Education, 44(1), 50-63. http://dx.doi.org/10.1111/j.1365-2923.2009.03547.x

Neufeld, V., & Norman, G. R. (1985). Assessing clinical competence. New York: Springer.

Oh, P. J., Jeon, K. D., & Koh , M. S. (2015). The effects of simulation-based learning using standardized patients in nursing students: A meta-analysis. Nurse Education Today, 35(5) e6-e15. http://dx.doi.org/10.1016/j.nedt.2015.01.019

Pierre, R. B., Wierenga, A., Barton, M., Branday, J. M., & Christie, C. D. C. (2004). Student evaluation of an OSCE in paediatrics at the university of the West Indies, Jamaica. BMC Medical Education, 4(22). http://dx.doi.org/10.1186/1472-6920-4-22

Selim, A. A., Ramadan, F. H., El-Gueneidy, M. M., & Gaafer, M. M. (2011). Using objective structured clinical examination (OSCE) in undergraduate psychiatric nursing education: Is it reliable and valid? Nurse Education Today, 32(3), 283-288. http://dx.doi.org/10.1016/j.nedt.2011.04.006

Singapore Nursing Board. (2012). Core competencies of a registered nurse. Retrieved
from http://www.healthprofessionals.gov.sg/content/dam/hprof/snb/docs/publications/Core%20Competencies%20and%20Generic%20Skills%20for%20RN_SNB_1%20Dec%202012.pdf.

Wind, L. A., Van Dalen, J., Muijtjens, A. M. M., & Rethans, J. (2004). Assessing simulated patients in an educational setting: The MaSP (Maastricht Assessment of Simulated Patients). Medical Education, 38(1), 39-44.
http://dx.doi.org/10.1111/j.1365-2923.2004.01686.x

 

APPENDIX. SIMULATION-BASED ASSESSMENT TOOL (SAT)


The Simulation-based Assessment Tool (SAT) focuses on the six core competencies expected of a Registered Nurse.

The SAT is used in conjunction with a case scenario in any setting in which the student nurse will be attending to a simulated patient.

From the performance, the assessor is required to rate each of the six core competencies from unsatisfactory to outstanding on a scale ranging 1 to 9. A rating of 4 is defined as “satisfactory” and implies that with supervision, the student nurse will meet the competencies expected at his/her level of training. All six core competencies should be rated 4 or above and a total score of 27 or more to consider a passing grade.

 

DESCRIPTORS OF COMPETENCIES DEMONSTRATED

Critical Thinking:

  • Identifies the rationale for performing the nursing care
  • Interprets baseline findings
  • Determines appropriate method and device, if needed, to perform nursing care
  • Reflects during performance phase
  • Evaluates the findings
  • Makes decisions after reflecting on intervention

 

Communication

  • Identifies any communication barrier / special needs
  • Assesses patient’s understanding of their condition and the procedure
  • Demonstrates ability to provide clear explanation of the purpose and the process of the nursing care
  • Applies the appropriate techniques to overcome any barrier and meet the patient’s needs
  • Provides appropriate intervention (reassuring, comforting, supporting, enabling)
  • Evaluates the patient’s understanding of information provided

 

Technical Skills

  • Performs the nursing care competently
  • Clears and disposes appropriately

 

Critical Thinking:

  • Identifies the rationale for performing the nursing care
  • Interprets baseline findings
  • Determines appropriate method and device, if needed, to perform nursing care
  • Reflects during performance phase
  • Evaluates the findings
  • Makes decisions after reflecting on intervention

 

Communication

  • Identifies any communication barrier / special needs
  • Assesses patient’s understanding of their condition and the procedure
  • Demonstrates ability to provide clear explanation of the purpose and the process of the nursing care
  • Applies the appropriate techniques to overcome any barrier and meet the patient’s needs
  • Provides appropriate intervention (reassuring, comforting, supporting, enabling)
  • Evaluates the patient’s understanding of information provided

 

Technical Skills

  • Performs the nursing care competently
  • Clears and disposes appropriately

 


 



About the Lead Author

TAN Khoon Kiat is Manager (Projects) and Senior Lecturer at the School of Health Sciences (Nursing) at Nanyang Polytechnic and was previously part of the academic team at ALCNS when the project was conducted. His research interests are in community health of older people and education. He has published in several journals, including Journal of Advanced Nursing and Patient Education and Counseling.

About the Corresponding Author

LIAW Sok Ying is part of the academic team at ALCN who is a registered nurse. She is an Assistant Professor with research interests in quality and safety in healthcare including the use of simulation training to build competencies in the healthcare workforce. She has published over 30 peer-reviewed papers in international refereed journals and presented her research at over 30 international conferences.


0 Reply

Reply to this Discussion

Do you have a view you'd like to share?

Name*

Email*

Comments*

Verify*

capcha