Case study: Chartered Accountants Ireland (CAI)
Debuting a digital exam platform to the time-honored establishment of Chartered Accountants Ireland was nothing short of a monumental task, but Cirrus did not shy away from the challenge.
Faced with the rising costs of testing centers and the fast-paced demands of a world going digital, Chartered Accountants Ireland knew they needed to make an enterprising decision to transform their institution. CAI is a prestigious organisation founded in 1888 and is the third oldest accountancy body in the world, dating back to before the Irish republic. They were traditional in the way they approached assessments, paper was always the way exams were taken.
Cirrus signed the contract with the Chartered Accountants in January 2019, with the deadline of delivering their first exams at the end of March, with the additional need to develop two bespoke question types for them. Cirrus has never backed down from a challenge and started immediately with gathering a team of experts for CAI. Cirrus understood that there was much at stake for CAI, both reputational and economic, and therefore when successfully implemented it could mean the complete metamorphosis from old tradition to the digital age.
Chartered Accountants Ireland (CAI) was established by Royal Charter on 14 May 1888, and is Ireland’s largest accountancy body. It represents more than 25,000 members globally. Qualified members of the Chartered Accountants Ireland earn the designation ACA.
In the autumn of 2018, CAI did a tender process to select a provider for both remote proctoring and end-to-end e-assessment platform. After a thorough process, the choice fell on Cirrus and ProctorU. The partnership stood out from the rest by being innovative and user-friendly and give CAI the confidence that they would be well taken care of.
Chartered Accountants Ireland is a prestigious organisation dating back to before the Irish republic and were traditional in the way they approached assessments. But the costs of test centres forced them to look at other venues for exam delivery. Together with their sister organisation they deliver in excess of 35,000 exams a year.
Cirrus signed the contract with the Chartered Accountants in January 2019, with the deadline of delivering their first exams end of March. And that with the need to develop 2 bespoke question types for them. Cirrus has never backed down from a challenge and started immediately with gathering the team for CAI. The training has been done for CAI and our consultants are working with them to implement the platform. At the same time, a team of developers are working on the new question types. At Cirrus we understand that it’s much at stake for CAI, both reputational and economic. We have therefore given a guarantee that the implementation and bespoke development will be ready in time.
Chartered Accountants Ireland was heavily dependent on paper-based assessments to successfully examine students whilst undergoing the ACA qualification at all levels.
Chartered Accountants Ireland’s desire was to migrate all of its existing paper based examinations, roughly 25,000 paper based assessments per annum, to an E-Assessment platform on a phased basis over the next three to five years. A pilot program was to be rolled out covering 10% of the exams but due to COVID-19, lockdown measures had to be implemented in March and pushed the entire project into overdrive. What started as a pilot program covering 10% of exams meant to slowly roll out over the course of five years now needed to be shortened to five months, covering ALL exams. Testing centers were closing and they needed to move quickly on a solution.
The challenge was tremendous, but if implemented properly, this new assessment platform will assist in reducing existing examination venue capacity issues, reducing workload relating to GDPR, increasing global reach of the ACA qualification and enhancing student satisfaction.
Planning and Implementation
The bond between Cirrus and Chartered Accountants Ireland began immediately as the project commenced. It was recognized that for the rollout to be successful, multiple stakeholders would need to be involved in the process and commit to working together to make this dream a reality.
- Identifying and mapping IT competencies
- Organizing a “kick-off” for all users involved
- Training of participants in project teams, users in need of extra IT support, super-users and administrators
- Individual coaching / assistance for project participants via Internet / Phone.
- Frequent meetings of project group / management committee.
- Discussion panels in which participants exchange experiences, show progress and
achievements, and discuss any challenges.
- Project completion including a final evaluation of the project
A detailed blueprint plan was devised to tackle these challenges and activities proposed to assist with transition:
- Detailed strategy sessions where Cirrus designed exactly the type of high-stakes exam questions CAI needed – including two new bespoke question types.
- Training programmes for staff including initial training and refresher workshops throughout the year. These would focus on each delivery phase (item creation, assessment creation, pilot delivery, marking and moderation).
- Chartered Accountant Ireland and Cirrus Assessment would together outline plans, provide updates, demonstrate functionality and answer questions and concerns in open forums.
The success of this transformation was due, in large part, to Cirrus truly listening to exactly what this accountancy body needed. It not only transformed Chartered Accountants Ireland from traditional paper-based to digital, but in turn also transformed Cirrus from a generic exam platform to one capable of delivering the most complex financial questions in the world.
Being that CAI is the third oldest accountancy body in the world and has a dignified reputation and position of prestige to uphold, taking the tradition of sitting exams the old-fashioned way in exam halls with pen and paper and introducing a new software to revolutionize their organization was a big task.
Not only was there a short time-frame to develop bespoke functionalities, but the high-stakes nature of the assessments CAI is tasked to give made this project especially complex. Chartered Accountants Ireland are guarantors of their Member’s reputation and it was an unquestionable must that Cirrus needed to be able to deliver exams with security in mind. There had to be no risk of any leakage of CAI data or of 3rd parties, accidentally or otherwise, coming into possession of any CAI data. Chartered Accountants Ireland are also Data Controllers and process their Member and Student’s personally identifiable data on their servers. Cirrus is a Data Processor within the meaning of the General Data Protection Regulations and had to satisfy Chartered Accountants Ireland as to their ability to secure CAI’s data and to meet their responsibilities under the GDPR. Cirrus was able to go above and beyond the requirements of what CAI needed in terms of GDPR in that no data tracks back to the student after the exam has been taken.
- 100% of CAI high-stakes exams online in 2020;
- Full regulatory approval that exams were as secure as paper-based exams in exam halls
- Annualized pass rates at all levels being completely consistent with the prior 3-5 years;
- Quality of responses from candidates was stronger than in prior years as students could type more within the time allowed and the formatting functionality facilitated clearer, more concise and professional report responses. This resulted in higher average marks than prior paper-based outcomes;
- Candidates had the ability to have 5 practice exams to familiarize themselves with the new platform. This would not have been feasible in an exam hall;
- Candidates all had access to a practice “onboarding” session to ensure that their hardware and broadband met requirements one month prior to the actual exam. This helped to facilitate the 100% real exam day onboarding success rates, which are completely unrivalled in the high stakes global professional accounting exams market (worldwide onboarding success rates ranged between 70-95%);
- Streamlined, intuitive end-to-end marking within the Cirrus platform resulted in exam marking times being reduced by up to 50%. This facilitated the earlier release of exam results in 2020, resulting in significant resource, time, and cost savings in 2020 and beyond;
- The very efficient marking process has resulted in greater satisfaction levels within our internal teams
- Quicker exam results feedback for students, heightening satisfaction
- Reduced carbon footprint
- Cirrus’ flexibility assisted CAI in existing customer retention and facilitated growth into other markets.
- The success of the project has given CAI a reputational boost. IFAC published a positive case study stating how CAI is at the forefront of digital exams on the global accounting stage.
- FRC and IAASA have both taken the CAI/Cirrus journey as a ‘best in class’ experience in 2020, enhancing CAI’s reputation.
From the very beginning Cirrus and CAI were an unlikely pair – a venerable establishment and a young, innovative online platform. In coming together with the goal of transforming CAI from paper-based to digital they discovered a perfect fit, in large part because of the willingness and flexibility of Cirrus Assessment in catering to CAI’s needs in every single way.
For CAI, the benefits of digital exams are numerous. They now have the ability to give exams to students all over the world in all circumstances. Tragedies like COVID-19 will never completely derail their students’ ability to take exams and continue prospering in their careers. Thanks to results being delivered more quickly due to digital grading, students are able to let accountancy firms know their results quicker, thus resulting in them being able to progress to a better payscale faster. CAI has also saved money on the administration costs of setting up, invigilating, and hosting exams in testing centers.
Perhaps the most unique aspect of the project was not only the enormous task of making the switch from paper to digital, but the condensed timeframe in which it was done within. This meant that absolutely everyone on both teams needed to come together as one united force in order to make the digital shift happen smoothly. This transformation would not have been at all possible had it not been for the communication, support, hard work, and open-mindedness of these two companies.
Thanks to the success of this cooperation, the E-Assessment project rollout in 2020 has given CAI a significant reputational uplift globally as IFAC published a very positive case study on its website stating how CAI are leading the way in this space on the global accounting stage.
Ease of use
CAI students found the system extremely intuitive to use as evidenced by the increase in average marks and the consistency in annualised pass rates. The student feedback post-exam was extremely positive. Only 2-5% of candidates actually experienced any difficulties, but these were mainly due to issues on their side, such as broadband connectivity, firewall issues, and poor hardware, not a Cirrus issue. Students love the rich text formatting options that they were familiar with. Also the ability to have pop up windows for resources, response windows or split screen capability was all welcomed too. CAI provided extensive online resources and practice papers to the students months in advance of the exam days as they could be completely familiar with the platform. This worked extremely well for all those that meaningfully engaged with the resources available to them. For students with reasonable accommodations, Cirrus was able to provide a wide range of solutions to help, such as read speaker, adjustable text size, and the ability to run a much larger second monitor (only allowable for visually impaired students). Where students had poor broadband in almost all cases students were able to sit their exam whilst hotspotting their laptop from their mobile phone, which made CAI exams accessible to absolutely everyone in 2020.
CAI provided administrators, examiners and markers with process documentation and video tutorials on how to use the Cirrus platform. These materials were designed to be as minimalistic as possible. Due to the extremely intuitive user interface for these stakeholders, all constituents were able to be highly functioning on the platform with minimal training. The platform was praised by all users as to how effective, efficient and fit for purpose it was. This was only achieved through huge collaborative work during Summer 2020 between Cirrus and CAI teams.