November Software Update: Further Enhancing Exam Fairness 

Share via:

Welcome to the November update blog, where we’re excited to share with you Cirrus’ most recent and upcoming enhancements. This past month’s enhancements are tailored to help you deliver fairer assessments and make your assessment process more efficient. From using Angoff to set passmarks, the introduction of Form Difficulty and the newly designed Forms Page, every change has been made with your needs in mind. Let’s dive in! 

Support for the Angoff Method

First up, we’ve introduced the option to use the Angoff Method to set passmarks. What’s special about it is that it adjusts the passmarks based on how difficult the exam is. So, if an exam is harder, the passcore will be set accordingly, making sure that the assessment is fair and balanced for everyone taking it. This is especially beneficial if you are using random forms. 

Quick recap: What is the Angoff Method? 

The Angoff method involves subject matter experts estimating the percentage of hypothetical test takers who would pass a test, considering their knowledge, skills, and question difficulty. Based on these estimates, a cut score is determined that will ensure a certain percentage of test takers pass the test.

Why use the Angoff Method? 

Increased Accuracy and Fairness

The Angoff method ensures pass marks are calibrated to the exam’s difficulty, leading to more precise and equitable assessments. 

Unseeded Questions

Involving subject matter experts in setting cut scores is particularly beneficial for questions that haven’t been tested or seeded yet. Their insights ensure that the pass marks accurately reflect the essential competencies required in the field, even for newly introduced questions.

Adaptability to Exam Complexity

The Angoff method’s adaptability is key in maintaining a consistent standard of assessment regardless of the exam’s complexity. It adjusts the cut score according to the exam’s difficulty, ensuring fairness and accuracy in diverse testing scenarios. This is especially beneficial when using random forms; it automatically adjusts cut-scores to ensure fairness, preventing any advantage from variations in question difficulty.

Simplified Passmark Calculation

Cirrus’ automatically calculates the passmarks, based on the Angoff values assigned to each item, streamlining the process of determining passmarks. This feature reduces the administrative burden and enhances the efficiency of the assessment process. 

How It Works in Cirrus

Using the Angoff method in Cirrus is simple and straightforward: 

  1. Activate the Angoff method in Cirrus by navigating to ‘Admin > Global settings’ and enabling it under the ‘Item difficulty scale’.
  2. Once activated, access the ‘Library > Collection > Statistics’ section to find the new ‘Angoff’ column.
  1. Enter Angoff values for each test item in the assessment, using up to three decimal places for precision.
  2. Ensure all questions in your assessment form have an Angoff value assigned to them.
  3. The system will automatically calculate and set the cut score and pass mark based on these Angoff values.

Learn more about using the Angoff Method in Cirrus here in our Knowledge Base:  

Introducing difficulty in forms

Next, we have introduced the ability to set desired difficulty levels for each form in Cirrus. This new feature allows you to specify the percentage of Easy, Medium, and Hard questions in both fixed and random forms.

This change significantly enhances Linear On the Fly Testing (LOFT). LOFT lets you deliver random, unique exams to each test-taker, which is highly effective against cheating. The key challenge has been maintaining consistent difficulty across different exam versions, and this new feature effectively addresses that concern.

What has changed?

Customisable Difficulty Levels

When creating a form, you can set the desired difficulty level using either P-values or Angoff values. Just as a reminder: P-values are automatically calculated by Cirrus each time a question is exposed to a candidate, while Angoff values are set by subject matter experts for items that haven’t been used yet.

Visibility of Difficulty Metrics

The Forms overview tab displays the average difficulty per form, giving you a clear picture of the overall challenge level.

Fairness in Random Forms

For exams using random forms, this feature ensures that every candidate receives a set of questions comparable in difficulty, maintaining assessment integrity.

Greater Fairness with the Angoff Method

Using the Angoff method for setting cut scores in conjunction with difficulty settings ensures even greater fairness. The method considers the difficulty of the assessment when calculating the cut score.

How It Works in Cirrus

Activating Difficulty Settings

Users can set the difficulty level during the form creation process, choosing between P-values and Angoff values based on their preference and the nature of the questions.

Monitoring Difficulty Levels

The average difficulty of each form can be monitored in the Forms overview tab, allowing for ongoing assessment and adjustments as needed.

Learn more about difficulty levels in Cirrus in our Knowledge Base.

Improved Forms Page

Last but not least, we’ve revamped the Forms Page, introducing a separate tab for Intro and Finish pages, a new Questions tab with enhanced functionalities, and a new table component for advanced filtering and search options.

What has changed?

Improved User Interface

We have separated sections into dedicated tabs to simplify navigation and enhance content management. You now have distinct tabs for the Intro, Finish, and Questions pages, making it easier to manage different sections of an exam.

Increased Efficiency and customisation

Advanced search, filtering and sorting options with the new table component make it much easier to manage your data. In the Questions tab, filtering and selecting the questions you’d like to use for each form has greatly improved. Moreover, users can customise the columns of the new table component to fit their specific needs, enhancing the visibility and accessibility of important information.

In-Depth Insights

Two new columns have been added for deeper insights into candidate performance and question relevance, aiding in better exam preparation and analysis.

  • Last Answered: This column displays the date when a candidate last answered a specific question, providing valuable data for tracking progress and question effectiveness.
  • Topics: Building on the Learning Objectives feature, the Topics column offers additional categorisation, aiding in the organisation and analysis of assessment content.
Angoff Method to set cut-scores

In both the Forms and Scale pages, we’ve integrated support for the Angoff method to set cut-scores. This addition offers a more sophisticated approach to determining passmarks, enhancing the fairness and accuracy of assessments.

Looking ahead

Improved Library

A complete overhaul of the tables managing Collections, Items, and Media elements is on the horizon. This update is designed to ensure a consistent user interface throughout the application, enhancing usability and coherence.

Enhanced Manual Question Selection

Upcoming releases will also bring improvements to manual question selection, giving users greater control and precision. It will include a feature for a read-only view of learning objectives, taxonomies, or topics of selected questions, aiding in ensuring the correct balance of items across different disciplines.

Reasonable Adjustments Feature

We are excited to introduce a new feature that allows for customising the percentage of extra time allocated per user. This development is geared towards accommodating diverse user needs, enabling a more flexible and inclusive assessment experience.

In conclusion, the November software update from Cirrus brings significant advancements in assessment fairness and efficiency. The integration of the Angoff Method for passmark setting, the introduction of form difficulty levels, and the redesigned Forms Page collectively enhance the precision, fairness, and user experience of our e-assessment platform. These improvements underscore our commitment to evolving with the needs of educators and assessment professionals. 

Looking ahead, we are excited to continue this momentum with further enhancements that will streamline the assessment process and cater to a diverse range of user needs. Stay tuned for more updates as we strive to make your assessment experience as effective and equitable as possible.

Any questions? Please don’t hesitate to contact your Customer Success Managers

Share via:
Topics
Picture of Vegard Sivertsen
Vegard Sivertsen
Cirrus’ founder and CEO is passionate about the potential of EdTech to transform how we teach and assess skills and knowledge.
Would you like to receive Cirrus news directly in your inbox?
More posts in Product Updates
Product Updates

Seizing Opportunities in Assessment: Cirrus’ Vision for Q3-Q4 2024

At Cirrus, our top priority is to help you seize new opportunities in assessment and overcome challenges. From AI Item Generation and AI Essay Marking to authentic, skill-based evaluations, Cirrus is not just adapting to these changes; we’re leading the charge. Find out how our bold plans for Q3-Q4 2024 will revolutionise your assessment processes, making them more efficient and effective.

Read More »
 

Curious about all things e-assessment?

As Cirrus looks to the future, we are excited to bring you the latest news, trends, and useful information about the industry.

 

Subscribe to the monthly Cirrus Examiner to join our ever-growing community of people passionate about the unbridled potential of EdTech.