I’m sure standardised exams are something we’re all (albeit begrudgingly) familiar with, but why have them at all? Well, for starters, if you’re ever laying on a surgeon’s table, sitting in a courtroom, or hiring an accountant, we’re betting you want a doctor, lawyer, or accountant that is guaranteed to be a shining example of expertise in their field.
Standardised exams are essential in a professional setting because they provide a standard benchmark for evaluating the knowledge and skills of employees, ensuring competent workers, a safe public, and a functioning society. In short, standardising exams are essential for your business or awarding body.
Benefits of the standardised exam
Let’s have a look at the benefits of the standardised exam. It can be used to:
Benchmark test-taker performance
Standardised exams are designed to be administered in the same way to all test-takers, so they provide a way to compare the performance of test-takers from different offices, districts, and even countries. This allows trainers to see how their employees are doing compared to others and identify areas where they may need additional support.
Identify strengths and weaknesses
Standardised exams provide a comprehensive overview of a test-taker’s knowledge and skills, and can help educators identify areas where a test-taker is excelling and areas where they may need additional support. This information can be used to tailor instruction and provide targeted support to help test-takers succeed.
Hold companies accountable
Standardised exams can be used to evaluate the effectiveness of a training program. By comparing the performance of students from different programs, policymakers and trainers can determine which approaches are most effective and make adjustments as needed to improve test-taker outcomes.
Identify gifted and talented employees
Standardised exams can be used to identify trainees who are excelling in specific subjects or areas, and can help educators provide additional support and resources to help these test-takers reach their full potential.
Inform curriculum development
The results of standardised exams can inform curriculum development and identify areas where test-takers may be struggling. This can help educators create more effective instructional materials and improve test-taker outcomes.
Overall, standardised exams are an essential tool for evaluating test-taker performance and helping to ensure that all test-takers have the knowledge and skills they need to succeed in their studies and life.
One of the methods to achieve standard setting is the Angoff Method. Let’s dive deeper into what this is:
What is the Angoff Method?
The Angoff method is a statistical technique used to set cut scores, the minimum scores required to pass a test or evaluation. It is named after William H. Angoff, who developed the method in the 1970s.
The basic idea behind the Angoff method is to ask a group of subject matter experts to consider a hypothetical group of test takers and estimate what percentage of those test takers would be able to pass the test if they were to take it. Those experts are asked to consider the test takers’ level of knowledge and skills and the difficulty of the test questions. Based on these estimates, a cut score is determined that will ensure a certain percentage of test takers pass the test.
The Angoff method is often used to set cut scores for professional licensing exams, such as the bar exam for lawyers or the medical licensing exam. It is also used in education to set passing scores for high stakes tests, such as state-mandated assessments.
One advantage of the Angoff method is that it involves the input of subject matter experts who deeply understand the content and skills being tested. This can help ensure that the cut score is fair and reflects the knowledge and skills necessary for success in a particular field. However, the Angoff method also has some limitations, such as the subjectivity of the experts’ estimates and the potential for bias in their evaluations.
How does the Angoff Method work?
The Angoff method involves several steps. First, the test developer selects a panel of subject matter experts who are representative of the examinee population for the test. The panel members should thoroughly understand the content and skills the test intends to assess.
Next, the panel members are given a set of test items and asked to evaluate each item’s difficulty. They are asked to consider the level of difficulty and the content of the items and to assign a probability based on their expert judgment of the likelihood that a minimally qualified examinee would correctly answer the item.
Once the panel members have assigned probabilities to all test items, each item’s mean probability score is calculated. This mean probability score represents the panel’s consensus judgment of an item’s difficulty.
The final step in the Angoff method is to use the mean probability scores to set the cut score for the test. This is done by calculating the total number of points needed to pass the test, based on the mean probability scores of the items. The cut score is then set at the point where a minimally qualified examinee is expected to correctly answer a sufficient number of items to pass the test.
Advantages of the Angoff Method
Angoff has a lot going for it. Let’s have a look at some of the key advantages:
- Based on expert judgement: The Angoff method relies on the expertise and knowledge of a panel of subject matter experts to determine the cut scores for a test. This means that the cut scores are based on the expectations and standards of experts in the field, rather than on statistical analysis alone.
- Allows for subjective decision-making: The Angoff method allows for subjectivity in the decision-making process, which can be useful in cases where there may not be a clear cut-off point based on statistical analysis alone.
- Allows for the incorporation of test-specific considerations: The Angoff method allows the panel of experts to consider specific factors that may be relevant to the test in question, such as the difficulty of the test, the importance of the material being tested, and the consequences of incorrect answers.
- Widely used: The Angoff method is widely used in the field of testing and assessment, which means that it is well-established and has a long track record of success.
- Can be used with a variety of tests: The Angoff method can be used with a variety of different types of tests, including multiple-choice tests, essay tests, and performance assessments.
Disadvantages of the Angoff Method
There are also several disadvantages to the Angoff method, including:
- Subjectivity: The Angoff method relies on the subjective opinions of experts, which can vary widely. This can lead to inconsistency in the cut scores that are determined using the method.
- Time-consuming: The Angoff method can be time-consuming, as it involves gathering a panel of experts and having them individually estimate the difficulty of each test item.
- Limited generalisability: The Angoff method may not be generalisable to all populations, as the experts used to determine the cut scores may not represent the broader population being tested.
Despite these challenges, the Angoff method is widely accepted and reliable for setting cut scores on high-stakes exams. It is based on expert judgment and considers the test items’ content and difficulty, making it a valid and accurate approach to setting cut scores. It is also flexible and can be customised to meet an organisation’s specific needs. It has been widely adopted and validated in the field of professional assessment.
It is especially useful for setting cut scores on new, untested questions. Once questions have been asked and answered by enough test-takers to reach statistical significance, the cut score can also (or instead) be informed by the Pc-value.
As we grow, Cirrus is always looking to implement new techniques and methods that help our customers create even better assessments. Keep your eyes peeled for announcements about the inclusion of the Angoff Method into our platform!
With this feature, our users can streamline the cut-score setting process and increase its efficiency, accuracy, and transparency. A user-friendly interface will allow experts to enter their estimates, and provide a system for aggregating and analysing data, and setting final cut-scores based on the expert estimates. We believe implementing the Angoff Method will significantly benefit our users, and we can’t wait for you to try it out!
