Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination
Abstract
Background: One of the methods used for standard setting is the borderline regression method (BRM). This study aims to assess the
reliability of BRM when the pass-fail standard in an objective structured clinical examination (OSCE) was calculated by averaging the
BRM standards obtained for each station separately. Materials and Methods: In nine stations of the OSCE with direct observation
the examiners gave each student a checklist score and a global score. Using a linear regression model for each station, we calculated
the checklist score cut-off on the regression equation for the global scale cut-off set at 2. The OSCE pass-fail standard was defined as
the average of all station’s standard. To determine the reliability, the root mean square error (RMSE) was calculated. The R2 coefficient and the inter-grade discrimination were calculated to assess the quality of OSCE. Results: The mean total test score was 60.78. The OSCE pass-fail standard and its RMSE were 47.37 and 0.55, respectively. The R2 coefficients ranged from 0.44 to 0.79. The inter-grade
discrimination score varied greatly among stations. Conclusion: The RMSE of the standard was very small indicating that BRM is a reliable method of setting standard for OSCE, which has the advantage of providing data for quality assurance.
Key words: Borderline regression method, objective structured clinical examination, reliability, standard setting
reliability of BRM when the pass-fail standard in an objective structured clinical examination (OSCE) was calculated by averaging the
BRM standards obtained for each station separately. Materials and Methods: In nine stations of the OSCE with direct observation
the examiners gave each student a checklist score and a global score. Using a linear regression model for each station, we calculated
the checklist score cut-off on the regression equation for the global scale cut-off set at 2. The OSCE pass-fail standard was defined as
the average of all station’s standard. To determine the reliability, the root mean square error (RMSE) was calculated. The R2 coefficient and the inter-grade discrimination were calculated to assess the quality of OSCE. Results: The mean total test score was 60.78. The OSCE pass-fail standard and its RMSE were 47.37 and 0.55, respectively. The R2 coefficients ranged from 0.44 to 0.79. The inter-grade
discrimination score varied greatly among stations. Conclusion: The RMSE of the standard was very small indicating that BRM is a reliable method of setting standard for OSCE, which has the advantage of providing data for quality assurance.
Key words: Borderline regression method, objective structured clinical examination, reliability, standard setting