The University of Pretoria (UP) needed a new online assessment platform to replace their end-of-life summative assessment platform. They began the search for a new cloud-based assessment tool that had to satisfy strict requirements. It had to be 1) Stable and secure enough to let them deliver a large number of exams anywhere, anytime. And 2) Full-featured and innovative, but user-friendly at the same time.
They found that the Cirrus Assessment platform satisfied these requirements, which led to a close partnership between the two organisations. After only 7 months of implementation, 67 lecturers were already using the system, with two thirds of them working partly or fully autonomously. During this time, 265 assessments were compiled and delivered to 17,307 students in the computer laboratories on campus.
The University of Pretoria’s assessments gained the following advantages from the new platform and partnership with Cirrus:
- Students can now take high-stakes assessments anywhere, anytime.
- Lecturers have increased autonomy over their assessments and have direct access to the statistics in the system to improve the quality of their assessments.
- The entire assessment process has become more streamlined and efficient due to a single data bank on the Cirrus’ cloud-based solution, improved blueprints, workflows, the impersonation function and system integrations.
“Thanks to the close partnership with Cirrus, we were able to achieve a much better aligned e-assessment process than we had before. We valued their knowledge, friendly approach and willingness to support us during the process to find solutions for our environment.
“With its streamlined assessment workflow, Cirrus has improved the everyday functioning of our team considerably. And the increased autonomy of our lecturers lets them focus on that for which they are appointed: improving the quality of their assessments.“
Detken Scheepers – Head of e-Learning, University of Pretoria
For this case study, we interviewed Ms Detken Scheepers (Head of e-Learning), Dr El-Marie Mostert (Education Consultant & E-learning Project Manager) and Dr Wimpie Beeken (Senior Project Manager), from the University of Pretoria.
Background
About the University of Pretoria
Located in South Africa, the University of Pretoria had humble beginnings with only 32 students in 1908. Present day, they have grown to six campuses serving over 50,000 students with 1 300 study programmes. As one of the top five universities in South Africa, they are proud of their 267,000 alumni.
“From humble beginnings in 1908, the University of Pretoria has grown to be one of the top universities in South Africa, with over 50,000 students.”
Dr Wimpie Beeken – Senior Project Manager, University of Pretoria
The University of Pretoria uses a combination of assessment strategies which include various e-assessment solutions, to address the needs of all the disciplines in the institution. High stakes invigilated electronic assessments have been conducted in a dedicated e-assessment system within computer laboratories across 5 campuses since 1993.
The previous invigilated e-assessment process
The previous process was inflexible and admin-heavy:
- Lecturers were able to develop only basic question types in a cloud-based application. The staff who supported lecturers with computer-based assessment (known as the CBT team) had to transfer these questions to the production environment and set more advanced question types for the lecturers. This resulted in lecturers not having direct access to their full question banks.
“We have used e-assessment software since 1993. Before Cirrus, we mainly conducted invigilated e-assessments in our computer labs, but it was a complicated process with a heavy administrative burden.”
Dr El-Marie Mostert – Education Consultant & E-learning Project Manager, University of Pretoria
- Assessments were compiled by the CBT team according to the preferred settings of the lecturer.
- Then, a proofreading session was scheduled with the lecturer to confirm that the questions and settings were correct.
- For assessments to be scheduled, student class lists had to be downloaded from the administration system and uploaded into the assessment system.
- Assessments were all taken by the students in the computer laboratories, limiting the number of assessments that could be delivered simultaneously.
- After completion, all results and reports were exported by the CBT team and provided to the lecturers.
Challenge
The process described above presented challenges as it became increasingly troublesome for lecturers who had little autonomy and control over the entire assessment process. The increased demand for this type of assessment led to a heavy administrative burden on the CBT team that became rapidly impossible to meet.
When the legacy system reached its end of life and the Information Technology infrastructure at the university was upgraded, UP used the opportunity to investigate a cloud-based solution.
UP’s requirements for a new online assessment system
In order to streamline their invigilated e-assessment process, UP formulated a number of requirements that had to be met by their new e-assessment software:
- First and foremost, they needed a stable and secure platform that could enhance the use of online assessment and enable students to take assessments not only in the computer laboratories but also from other venues.
- Additionally, they were seeking to increase the autonomy of the lecturers to compile their own assessments.
- The ability to share a variety of robust, innovative question types with colleagues and build question banks that can be reused was very important.
- It was essential that the new system would allow the transfer of the large number of existing question banks into the new platform. This necessitated forming a close partnership between Cirrus developers and UP’s staff.
- Considering how disruptive it would be to replace an online assessment system, UP also required the new system to be sustainable for the long term. They wanted e-assessment software that had a clearly defined roadmap and could grow with them into the future.
“We needed a sustainable system with a clearly defined roadmap and vision for the future as it would be a huge undertaking to replace our online assessment system.”
Dr El-Marie Mostert – Education Consultant & E-learning Project Manager, University of Pretoria
- The system should also comply with National and International laws as well as the University’s Information Governance and Privacy Protection (iGaPP) programme.
- Lastly, integration with their Student Administration System and LMS would be regarded as added benefits.
Solution
Functional pilot
As the CBT team’s initial investigation into the Cirrus Assessment system showed promise, the university decided to conduct a more in-depth pilot project with lecturers and students to determine if the product would address the majority of their assessment needs.
The pilot lecturers conducted formative assessments from September to mid-November 2020 in seven faculties. Students participated in the assessments using their own devices off-campus, with a few assessments completed in a computer laboratory on campus.
Based on the feedback received from the pilot participants, the steering committee of the pilot project then decided to implement the system.
Implementing a new e-assessment tool
Dr Wimpie Beeken (Senior Project Manager) developed a detailed and exacting project plan to manage the collaboration between the Cirrus Implementation Team, and the various UP teams. The plan followed the following process:
- Review the outcomes of the functional pilot project as input to the User Requirements Specification (URS) document.
- Complete a detailed URS and identify additional functionalities to be developed.
- Prepare a hosted environment based on UP’s enterprise standards to include Development, Testing, Production and Training Environments.
- Develop a migration tool to migrate question banks to Cirrus.
- Develop, test and sign-off additional functionalities.
- Create a targeted communication and training strategy to prepare users.
“From a project perspective you must ensure a comprehensive pilot project – both technical & functional – to understand the full scope and cost of the implementation.
“Next, having an implementation partner who is very involved in shaping the project objectives and willing to ‘walk-the-talk’ is also vital. This was the case for our project, where Cirrus Founder & CEO Vegard Sivertsen was present and willing to resolve project-related matters with us.“
Dr Wimpie Beeken – Senior Project Manager, University of Pretoria
Aligning Cirrus to UP’s internal processes
It was necessary to align UP’s internal processes with the workflows in Cirrus. UP and Cirrus held extensive consultation sessions and compiled a user specification document to record decisions and specifications for new developments. After consultation across the university, UP also updated the roles and responsibilities document that governs the activities of the academic – and support departments.
Developing new features together
The consultation process revealed a few needs of the university that were not addressed by the system. UP and Cirrus collaborated to improve existing features to better align the platform to UP’s internal policies and ways of work. This included an update to the blueprint functionality, enabling assessors to auto-generate assessments based on question selection from topics.
Migrating 200,000+ exam questions
The migration of more than 200,000 existing questions from 5 different databases to the Cirrus platform posed a challenge to both UP and Cirrus. Engineers from Cirrus developed a migration tool to assist UP with the transfer of the questions. The CBT team turned this challenge into an opportunity by collaborating with UP lecturers to clean up the data. This helped UP achieve a reduction of 60% in storage space.
“We turned the challenge of migrating all our old assessment data into an opportunity: By consolidating it on the Cirrus cloud platform, we achieved a 60% reduction in storage space.”
Detken Scheepers – Head of e-Learning, University of Pretoria
Combining the data into one database also made it possible to implement several integrations of the Cirrus platform with:
- The UP Student Information System: This enables automated user management and a user-friendly single sign-on (SSO).
- The university’s Learning Management System (LMS): This allows the sharing of student results between Cirrus and the LMS. This greatly reduces the administrative burden and associated potential errors.
Preparing the users
After solving the technical challenges, the last and arguably, biggest, task of the implementation was managing the move to a new platform with UP’s various users. For this reason the University of Pretoria promoted the Cirrus e-assessment platform via emails, conference presentations, a marketing video, and a well-developed training programme that is presented regularly. The CBT-team also provides personal support to users throughout.
“The biggest task of a new software implementation comes after solving the technical challenges: Managing the change with your users.”
Detken Scheepers – Head of e-Learning, University of Pretoria
Results
Delivering assessments the UP way
UP lecturers can now manage their question banks and assessments independently, and although the CBT team is still very much involved with training and support, the lecturers are empowered to work autonomously in the system. Students can take high-stakes assessments off-campus and the remote invigilation functionality lets lecturers monitor student activity and connectivity.
“Our lecturers are now empowered to work autonomously and students can take high-stakes assessments off-campus.“
Detken Scheepers – Head of e-Learning, University of Pretoria
Easier and less time-consuming assessment processes
The workflows enabled by Cirrus streamlined various processes for UP:
- Moderation of questions are accommodated easily and securely through the in-app sharing of the collections.
- Labels make it easier for lecturers to find questions in the database.
- Lecturers can compile assessments using the blueprint functionality that allow users to auto-generate assessments according to Topics and marks or Learning objectives and Bloom’s Taxonomy.
- Marking coordination among multiple assessors is easy to achieve.
- The integration with the student administration system and the Grade center of the LMS has made life much easier for lecturers and the CBT team alike.
- It has improved the everyday functioning of the support team considerably as the ‘impersonation function’ and workflows resulted in fewer emails between lecturers, assessment staff and the computer laboratory staff.
The benefits of a close partnership
For UP, the partnership with Cirrus during their move to a new online assessment system enabled them to achieve better aligned e-assessment processes. Thanks to the intense collaboration during the implementation process, Cirrus was able to support UP when needed and propose solutions that fit their existing way of working.
Cirrus also made a number of changes to the platform to better meet UP’s requirements. Among others, this included improvements to accessibility features that improved the experience of students with visual disabilities.
Overall, assessments at the University of Pretoria gained the following advantages from the new platform and partnership with Cirrus:
- Students can now take high-stakes assessments anywhere, anytime.
- Lecturers have increased autonomy over their assessments and have direct access to the statistics in the system to improve the quality of their assessments.
- The entire assessment process has become more streamlined and efficient due to a single data bank on the Cirrus’ cloud-based solution, improved blueprints, workflows, the impersonation function and system integrations.
