MyDispense has many unique features and options that allow students to get additional practice, immediate feedback, and practice experience without physically practicing in a pharmacy. The assessment feature is one of the unique features from a faculty perspective. Assessment within MyDispense allow for faculty to provide both formative and summative assessment with significant customization to students. Our focus today will be on assessment feedback; specific information on how to create assessments is located within the MyDispense Guides.
One highly requested feature of MyDispense was to provide feedback for assessments. Remember, our focus is not how to create the exam (see guides above), but instead how you can customize the feedback so students can see their scores as well as areas of improvement post-exams. One important note: If you use this feature, it will be possible for students to see content within your assessment. If you use MyDispense for secure examinations and/or Objective Structured Clinical Examinations (OSCEs) that need to be protected from the students, I suggest not releasing the results and instead posting the result in another forum, such as your institution’s Learning Management System (LMS) (e.g. Blackboard, Moodle, Canvas, etc.).
Let me set the stage for how we use MyDispense Assessments at the University of Kentucky College of Pharmacy (UKCOP). This is definitely not the only way (or perhaps even the best way!), but I think it will give you a context for the assessment results below. We have a six-semester simulation sequence that spans from Fall in Professional Year 1 (PY1) to Spring Semester of Professional Year 3 (PY3). Students use MyDispense independently each semester—with orientation during the first semester—and progressive complexity as they progress through the curriculum. At the conclusion of each semester, MyDispense is one of multiple competency assessments based on objectives defined at the beginning of each simulation module. Typically, this is a 20-minute, two exercise (combination of validation, dispensing, and/or OTC based on the semester) where students will either be given a “needs improvement, pass, or high pass” based on their performance. Students have three total attempts to meet competency (true for any assessment in the simulation module) and students receive feedback after each attempt to learn and improve to meet competency before progressing to experiential rotations (both IPPEs as well as Professional Year 4 (PY4) APPE rotations) and the subsequent semesters’ coursework.
Regardless of how you’ve designed your assessments, once you’ve completed your assessment (creation/design, administration of the assessment, and marking the exam), you are presented with a screen in Exam Manager that is a summary of the assessment. Here’s an example from a recent assessment administered at our institution:
If you want students to be able to review their results, select the ‘Release Results’ button next to the Export Feature near the top of the page. Note this doesn’t release the results immediately but instead takes you to the screen where you can determine how you would like these results presented to your students. Faculty can release and unrelease results at any time making this a useful feature during an ‘exam review session’ allowing students to review during a specified window of time. As of the current version (MyDispense Version 6.0.0 Beta 25 USA), ‘Release Results’ are set to open a certain time (current or in the future), but require faculty to manually pull the results back from the students if results should not be available. So what happens once you click ‘Release Results’?
From the figure above, you can see that faculty can ‘check’ or ‘uncheck’ one, two, or three sections for each assessment to display: overall exam mark (final score), marks per exercises (if you had multiple exercises on your assessment), and the exercise criteria (individual pointed items within your assessment). For each of these, you can select one of 5 options: none, points, percentages, bands, and ranked.
Bands are what we use at our institution as they allow the most customization for how you want your results to be displayed to the students. They are entirely customizable (see ‘Label:’) but are limited to three sections (low, medium, or high). Our bands reflect similar categories that are the three competency sections described above (needs improvement, pass, and high pass). You can label these however you like!
As far as the results for MyDispense Assessments at UKCOP, we design each exercise to 30 total points—these can vary based on the exercise (e.g., OTC vs. Prescription, Multiple Validation Prescriptions) but they always equal 30 total points. This allows us to set the ‘pass’ point at 80%. Now remember, because we use this approach, students do NOT see how points are distributed (we didn’t want them to focus on ‘high-value’ areas of the prescription) but instead focus on the whole process and what areas are ‘needs improvement’. When using this design, you can set the points to such a way that any ‘critical items’ are worth more than 6 points (i.e. ≥20%) meaning if a student misses them, they will not hit competency. Other areas (e.g., superfluous questioning of the patient or physician) may be worth 2 or 3 points; this means a student can still pass with a small mistake, but multiple mistakes and/or any critical items will result in a ‘needs improvement’. All the point totals are for faculty-facing documentation to establish the cut-points for each of the bands above.
One important point I learned from experience is creating bands is that if you have the ‘Marks per each exercise criteria’ checked, the label will apply to each exercise criteria section displayed to the students. For example, if they typed the name of the patient correctly in a prescription exercise (and received 1/1 points; 100%), assessment feedback will display ‘High Pass’ by that specific criteria even if they didn’t pass the assessment. Regardless of how you label your bands, it is very important to communicate to your students regarding the overall pass line/critical points, what the criteria mean, and how you want the assessment information presented to the students. Before releasing to the students, I would suggest you create a test account using a nonstudent email and take the examination yourself. You can quickly release and unrelease the results to get a better picture of what your students are seeing and how to customize to your institution.
There are lots of combinations on how to do this, this is just one example! We will cover other assessment strategies in future blogs. Please post below how you all approach feedback to students and if there is anything else you’d like to see from the assessment. Thanks for reading!
Recent Comments