Anyone who’s been involved in the implementation of an online platform knows that it’s not an off-the-shelf solution. Success hinges on aligning the new platform with your organisation’s existing processes and your future vision of e-assessment. Challenges like user resistance and overlooked requirements can derail progress, leading to increased costs and missed opportunities. Addressing these issues early through thorough planning and stakeholder engagement is key to a smooth implementation.
In their presentation at the e-Assessment Conference in London, Dr Beeken and Ms Scheepers shared the 6 top insights gained from their successful roll-out of a new online assessment platform at the University of Pretoria (UP), adopted by their 43,000+ students.
About the University of Pretoria
The University of Pretoria was established in 1908. With only 32 students walking through the doors on the first day, today they have more than 43,000 enrolled students and are one of the top five universities in South Africa. It consists of nine faculties and a business school with a very strong international profile, tapping into a wide alumni network to foster building their own entrepreneurial skills locally.
The University of Pretoria’s high examination pass rate, at 82.4%, highlights the important role that annual assessments play in their educational framework and the effectiveness of their assessment strategy. Remarkably, the University of Pretoria has employed digital testing for more than 20 years.
The need for a new e-assessment platform
The University of Pretoria required a modern online assessment platform to replace their existing inflexible assessment platform. Their quest led them to search for a cloud-based tool that had to meet stringent criteria: The selected platform needed to be 1) robust and secure, enabling the delivery of a multitude of exams anytime and anywhere, and 2) feature-rich, innovative, and user-friendly.
The Cirrus Assessment platform emerged as the perfect match, meeting all specified requirements, both in terms of technical specifications and support. To implement the platform, the university and Cirrus Assessment formed a close partnership to tackle challenges and tailor the platform to precisely fit the University of Pretoria’s needs. This collaborative effort ensured a seamless integration of the platform with the university’s requirements. Within a mere 7 months of implementation, 67 lecturers were actively utilising the system, with two-thirds operating autonomously, contributing to the creation and delivery of 265 assessments to 17,307 students in on-campus computer laboratories.
So how did they do it? These are the 6 key insights learnt by the University of Pretoria team from their successful implementation:
Insight 1: Select the right stakeholders
- The first step is to build up the workframe of functional managers and primary customers of the project, including team leads, project managers and team members.
- Choose functional managers who have expertise across different areas of your assessment process.
- Make sure the platform’s primary users feel that they can influence the project, as well as their level of importance to the project.
For larger organisations it is common to have two sponsors for implementing the e-assessment platform. In the case of The University of Pretoria (UP), they had a business owner sponsor, the director of the Department of Education Innovation and a second sponsor, the CEO. The second sponsor took decisions about ICT matters to the Steering Committee.
“You have to search for functional managers who have expertise across the different areas of your assessment process so that you can mould or repurpose certain processes to either fit into a specific functionality, or you need to develop new functionalities. Having multiple functional managers also helps to strengthen your user requirements specification, which you need to also avail some time to.”Dr Wimpie Beeken – Senior Project Manager, University of Pretoria
In the presentation, Dr Beeken pointed out the importance of the primary customers, those people who will use the online assessment platform. It is vital that they feel that they can influence the project, as well as their level of importance. For example, the implementation team made UP’s disability unit a primary customer of the solution because they would help them test the inclusiveness of accessibility of the solution.
Insight 2: Articulate your user requirements
- Lay out your functional requirements (product features and user requirements) and non-functional requirements (product properties and user expectations).
- Don’t overlook non-functional requirements, which detail how the system should operate.
- Conduct a pilot project to evaluate (non)-functional requirements and make sure the set-up mirrors your actual production environment.
The next step in the implementation is to define the functional requirements, outlining what the system should accomplish.
“Within Cirrus Assessment, we had a world-class jewel with a number of product features, and that’s why we selected Cirrus Assessment. We looked at our user requirements to see if there were any gaps or any gaps in our e-assessment process. And, collectively with the functional managers, we identified any new functionalities that we needed to look into for development.”Dr Wimpie Beeken – Senior Project Manager, University of Pretoria
However, it’s crucial not to overlook non-functional requirements, which detail how the system should operate. This becomes particularly important when transitioning to cloud-based solutions, where factors like scalability and security, must be closely analysed and tested. Dr Beeken recommended conducting a pilot project to thoroughly evaluate and refine these aspects. Non-functional requirements are essential to ensure that when piloting a solution, it occurs within a controlled environment. It should mirror your actual production environment to avoid overlooking any critical aspects of the setup.
Insight 3: Involve stakeholder representatives in each phase
- Involve stakeholders from the start.
- With regard to the pilot phase: Make sure there are representatives from all faculties/departments; Try to find volunteers rather than assigning staff; Physically observe users working with the system; Involve high-level decision-makers.
- Have an open discussion about roles and responsibilities in your assessment process.
Ensuring active involvement of stakeholder representatives at every project phase is of utmost importance. The University of Pretoria’s implementation team promoted this engagement by already involving stakeholders during the initial establishment of the system evaluation criteria. Then, they compiled a comprehensive list of functional, technical, and legal requirements, emphasising the necessity of incorporating all these aspects during these initial steps. Failure to do so might lead to project halts during implementation.
The importance of involving stakeholders was evident during the data migration phase, where lecturers actively contributed to cleaning up data banks. This collaboration meant that only 40% of the questions needed to be transferred, optimising storage space and cost, and achieving a significant victory for the team.
From their pilot of the new e-assessment system, the UP team gained the following insights: First, it is important to have representatives from all faculties involved in the pilot. This helps stakeholders feel involved and ready for the changes to come. Secondly, volunteers tend to have much deeper internal motivation than those who are assigned and told to participate. They tend to experiment more and thus offer more valuable feedback on the system. Third, observing both lecturers and students actively working within the system holds substantial value. In UP’s case, it helped highlight issues that needed to be addressed during subsequent user training.
Finally, it is crucial to have knowledgeable decision-makers involved in the pilot phase, responsible for approving or rejecting system implementation phases. These decision-makers must comprehend the system’s impact on the entire university and its various facets.
Addressing roles and responsibilities during the assessment process is also highly important: Ensuring that everyone has a voice in decision-making promotes active participation during the implementation phase. This inclusive approach fosters better adherence to established processes, as individuals feel a sense of ownership from being involved in the decision-making process. Ongoing feedback from training sessions further refines and improves the approach for subsequent phases.
“For this project, we had to do a fully online pilot, which was not very easy. We could not really do a pilot situated within our labs because the students were not allowed on campus [during the COVID-19 pandemic]. But the lessons that we learned from our pilot was that you have to have representatives from all faculties. This actually is making our life much easier at this point because now people feel that someone from the tribe was involved with the pilot and the decision and they had an insight into what was coming.”
Detken Scheepers – Head of e-Learning, University of Pretoria
Insight 4: Communicate clearly to all stakeholders and future users
- Take a comprehensive approach, ranging from high-level institutional communication to personalised emails, to ensure that stakeholders at all levels are well-informed, engaged, and prepared for the upcoming changes.
- Personal email tends to be most effective in stimulating concrete user action.
Ensuring clear communication with all stakeholder levels is of paramount importance. The University of Pretoria created a Project Steering Committee for project governance. For institutional awareness, a top-down approach through deputy deans was utilised, passing information about upcoming changes to faculty members.
To enhance awareness, the University leveraged platforms such as their teaching and learning conference together with their institutional marketing. Additionally, they facilitated knowledge-sharing among lecturers, allowing them to discuss effective practices. To reach existing users at a personal level, the team employed personalised emails. These emails covered the following key aspects for future Cirrus users:
- The Why: Clearly articulating the reasons behind the new system implementation.
- What’s in it for Me: Highlighting the individual benefits and advantages for the recipients.
- Timelines and Processes: Providing a clear understanding of the timelines and the processes involved.
- Request Collaboration: Encouraging active collaboration from users and emphasising their role in the process.
- Skills Development Opportunities: Communicating opportunities for users to enhance their skills through the new system.
UP also created videos to communicate key benefits of the system, creating curiosity and engagement and offered workshops as a platform for hands-on learning and deeper understanding. They discovered that personalised emails were most effective in the university context, especially when requiring action from users.
Insight 5: Provide the necessary training & skills
- Organise various training formats for different groups of users until they become autonomous.
- Make sure the implementation team is available for post-training questions, until users build their confidence.
- Provide the necessary training at the time needed to implement new skills.
The University of Pretoria organised a separate type of training for each level of user, from top level system admins through to the students. They wanted their lecturers to be more autonomous in the use of the Cirrus online assessment platform, so they created very focused training resources for them, including asynchronous and live online sessions by the implementation team on Cirrus usage. Recordings of these sessions were made available for lecturers to review at their convenience. A help site, heavily reliant on the Cirrus knowledge base, was accessible to further support users. The implementation team placed a strong emphasis on continuous assistance, recognising the importance of being available post-training to bolster confidence and guide users toward becoming autonomous.
Ms Scheepers underscored the significance of post-training availability, stressing that ongoing support is crucial for building confidence and fostering users’ gradual transition to autonomy. Additionally, she emphasised the importance of timing in training, suggesting that it should be delivered just in time to prevent forgetting and the need for retraining if conducted too soon for users.
“For our training and our skills development, we rather use training that is specific for each level of user. So from the top system administrators through to our students, each of these teams had a separate type of focused training for them. Our lecturers, we really wanted to be more autonomous in the use of the system, we cannot do everything for them anymore. So we have a very focused training for them where we have created a course in Blackboard Learn with synchronous and asynchronous formats.”Detken Scheepers – Head of e-Learning, University of Pretoria
Insight 6: Smooth project management is crucial.
- Leverage dual role project management with your e-assessment platform provider, where they offer the in-depth knowledge of the platform and possibilities to address client needs and you bring the institutional and functional understanding.
- Use visual planning to track high-level progress during regular status meetings.
The University of Pretoria and Cirrus Assessment established a close collaboration for the dual project implementation. Cirrus Assessment contributed in-depth knowledge of the platform and its capabilities to address user needs. They possessed the necessary tools for a smooth and rapid implementation, coupled with the ability to track and forecast task delivery based on their experience implementing Cirrus in other organisations.
As Business Owner, the University of Pretoria leveraged their institutional and functional understanding to manage steering committee expectations, oversee the budget, define overall end-goals, and conduct team meetings. The university also ensured regular coordination among team project managers to minimise overlapping workstreams, managing a total of 25 work streams with various activities. A visual planning approach during project team meetings further helped keep the project on track.
Both Dr Beeken and Ms Scheepers further emphasised the benefits of a strong partnership. as the capabilities of the entire team and the strength of the partnership help shape the scope of the implementation and the value you create.
“[…] having Cirrus Assessment on board, with their technical skills, and their expertise on how to approach e-assessment, helping us to remodel and rethink our own e-assessment model and also add additional functionalities, this is the value of the partnership where their capability helps us to drive the scope of the project.”Dr Wimpie Beeken – Senior Project Manager, University of Pretoria
“The partnership between the University and Cirrus created a win-win situation for all of us. For us as a university, it was really a breath of fresh air to work with an assessment partner who listens to our requirements and works with us to address our needs. And Cirrus also gained from our partnership with these new and amended functionalities. So our success resulted in their success.”Detken Scheepers – Head of e-Learning, University of Pretoria
Interested in learning more? Find out more about the University of Pretoria’s implementation of the Cirrus online assessment platform here.
Seeking more insights on implementing an online assessment platform? Get in touch with us today for expert advice and explore how we can enhance your assessment strategies together.