Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests
Search in google:
In Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests, authors Gregory J. Cizek and Michael B. Bunch provide the only "go-to" reference on how to set standards on tests in education, licensure, and certification. This book is comprehensive in scope, practical in nature, and definitive in terms of cataloguing the essential conceptual and procedural fundamentals of setting performance standards. Key Features: Offers a detailed description of procedures: In a practical, straightforward, and virtually jargon-free style, this book begins with important foundational and conceptual information about standard setting followed by step-by-step procedures for all currently used methods, including Angoff, Item Mapping, Analystic Judgment, etc. In addition, a full chapter is devoted to future issues and research in standard setting. Enables readers to design, implement, and evaluate a standard setting study: Using this book, readers will be able to: select an appropriate standard setting method; schedule and plan implementation; identify and train qualified standard-setting participants; conduct the standard setting meeting; calculate one or more cut scores; and evaluate the success of the procedure they have implemented. Allows readers to adapt written materials to their situation: Examples are provided in widely available formats such as Excel, SPSS, .pdf and Word files so readers do not have to purchase specialized software or learn to use unfamiliar packages. The book includes downloadable forms, sample correspondence, checklists, data sets, and analytical software for performing calculations required for various standard setting procedures. Intended Audience: This is an excellent supplementary textbook for graduate courses such as Applied Measurement in Education; Psychometric Theory; Advanced Educational Measurement; Special Topics in Measurement and Evaluation; Seminar in Educational and Psychological Measurement in the departments of Educational Psychology, Measurement and Evaluation, Psychology, and others.
Preface xiFundamentals of Standard Setting 1Contemporary Standard Setting: An Enduring Need 5The Need to Make Decisions 6The Benefits of Standard Setting 8General Approaches to Standard Setting 9Current Contexts for Standard Setting 11What Is Standard Setting? 13Kinds of Standards 14Definitions of Standard Setting 14Policy Issues and Standard Setting 19Scoring Models 20Research on Standard Setting 22Rounding 23Classification Errors 25Item Scoring Criteria and Total-Test Performance Standards 29Conclusions 33Common Elements in Setting Performance Standards 35Purpose 36Choosing a Standard-Setting Method 41Performance Level Labels 44Performance Level Descriptions 46Key Conceptualizations 48Selecting and Training Standard-Setting Participants 49Providing Feedback to Participants 53Normative Feedback 54Reality Feedback 55Impact Feedback 56Professional Guidelines for Standard Setting 57Evaluating Standard Setting 59Conclusions and a Foreword 63Standard-Setting Methods 65The Nedelsky Method 69Procedures for the Nedelsky Method 70Alternative Procedures and Limitations 71The Ebel Method 75Procedures for the Ebel Method 75Alternative Procedures and Limitations 78The Angoff Method and Angoff Variations 81Procedures for the Angoff Method 82Procedures for Angoff Variations 87The Extended Angoff Method 87The Yes/No Method 88Alternative Procedures and Limitations 92The Direct Consensus Method 97Procedures for the Direct Consensus Method 98Alternative Procedures and Limitations 102The Contrasting Groups and Borderline Group Methods 105Procedures for the Contrasting Groups Method 106An Example Using the Contrasting Groups Method 108The Borderline Group Method 112Alternative Procedures and Limitations 113The Body of Work and Other Holistic Methods 117The Judgmental Policy Capturing Method 117The Dominant Profile Method 120The Analytical Judgment Method 121Summary Analysis of Three Holistic Methods 122Overview of the Body of Work Method 123Procedures for the Body of Work Method 124Training 125Rangefinding 125Pinpointing 129Calculating Cut Scores 129Interpreting the Logistic Regression Output 135An Application of the Body of Work Method 138Selecting Work Samples 139Training Participants 140Rangefinding 140Alternative Procedures and Limitations 148The Bookmark Method 155Overview of the Bookmark Method 157The Ordered Item Booklet 160The Response Probability (RP) Value 162Response Probabilities and Ordered Item Booklet Assembly-Rasch Model 162Response Probabilities and Ordered Item Booklet Assembly-2PL Model 167Directions to Bookmark Participants 172Calculating Bookmark Cut Scores 176An Implementation of the Bookmark Procedure 177Training 177Introducing the Ordered Item Booklet 179Round One of a Bookmark Procedure 180Obtaining Preliminary Bookmark Cut Scores 181A Caveat and Caution Concerning Bookmark Cut Scores 184Round One Feedback to Participants 185Round Two of a Bookmark Procedure 186Round Three of a Bookmark Procedure 187Alternative Procedures and Limitations 189The Item-Descriptor Matching Method 193Procedures for the IDM Method 194Alternative Procedures and Limitations 202The Hofstee and Beuk Methods 207The Hofstee Method 209Procedures for Implementing the Hofstee Method 209The Beuk Method 212Procedures for Implementing the Beuk Method 212Alternative Procedures and Limitations 214Challenges and Future Directions in Standard Setting 217Scheduling Standard-Setting Activities 219Scheduling Standard Setting for Educational Assessments 219Overall Plan 222Participants 225Materials 231Final Preparations 234At the Standard-Setting Site and Following Up 236Scheduling Standard Setting for Credentialing Programs 237Overall Plan 238Participants 242Materials 243Final Preparations 244At the Standard-Setting Site and Following Up 244Conclusions and Recommendations 246Vertically-Moderated Standard Setting 249The Interrelated Challenges 250A Brief History of Vertically-Moderated Standard Setting 253What Is VMSS? 254Approaches to VMSS 256Applications of VMSS 257An Illustration of VMSS Procedures 262The Assessment Context 262Preparing to Implement a VMSS Approach 263Training VMSS Participants 264Facilitating the VMSS Standard-Setting Meeting 265Vertical Articulation of Cut Scores 266Final Review, Adoption, and Conclusions 270Alternative Procedures and Limitations 271Standard Setting on Alternate Assessments 275The Unique Challenges of Alternate Assessment 276Necessary Conditions for Alternate Assessment Systems 278A Generalized Holistic Method 283Overview of an Application of the GH Method 285Procedures for Implementing the GH Method 286Conclusions and Discussion Regarding the GH Method 292Alternative Procedures and Limitations 293Special Topics and Next Steps 297Rounding 298Methods of Adjusting Cut Scores 299Deciding How to Incorporate Uncertainty 302Generalizability of Standards 306Decision Consistency and Decision Accuracy 307A Demonstration of Computing Decision Consistency and Decision Accuracy on Complex Tests 313Other Decision Consistency Procedures 315Summary and Future Directions 317Using Multiple Methods of Standard Setting 319Improving Participant Training 320References 323Glossary 333Author Index 341Subject Index 345About the Authors 351
