To Compare the Efficiency of ChatGPT and Bard in Medical Education: An Analysis of MCQ-Based Learning and Assessment

Abstract

Objective: This study aimed to compare the efficacy of ChatGPT and Google Bard as virtual tutors in supporting students across various levels of cognition in MCQ-based assessments in the field of Internal Medicine.

Methods: This cross-sectional study was conducted in the Department of Internal Medicine in col- laboration with the Department of postgraduate medical education from June 2023 to October 2023. A comprehensive collection of multiple-choice questions (MCQs) covering various aspects of Internal Medicine was compiled by the research team’s consensus. The items were systematically organized into chapters and further categorized based on cognitive complexity levels (C1, C2, and C3). The cho- sen MCQs were entered into separate sessions of both ChatGPT and Google Bard. The responses from each Artificial Intelligence platform were then compared with the corresponding answers in the designated MCQs book. Recorded responses were classified as accurate, inaccurate, or partially ac- curate.

Results: The ChatGPT exhibited an overall success rate of 64%, providing 199 correct responses out of 307 queries, of which 10 were partially correct. By contrast, Google Bard achieved an overall suc- cess rate of 58.95 %, yielding 181 correct responses out of 307 queries, where 16 were partially cor- rect. When stratified by cognitive complexity levels, ChatGPT demonstrated proficiency in solving C2 MCQs at a rate of 80%, whereas the performance rates for the C1 and C3 categories were 69% and 54%, respectively. In contrast, Google Bard displayed a 33% success rate in solving C2 MCQs while achieving success rates of 95% and 53% in the C1 and C3 categories, respectively.

Conclusion: The findings of this study suggest that ChatGPT is a more advantageous tool for stu- dents and medical educators than Google Bard. These discerned advantages underscore the poten- tial of ChatGPT to enhance the educational experience within the medical domain.