Case study: National Council for the Training of Journalists (NCTJ)
The introduction of e-assessment by the National Council for the Training of Journalists was a huge step for the organisation and for the journalism industry as a whole. Faced with growing challenges around the authenticity and security of their paper based exams and demands from a fast changing digital media industry, the NCTJ needed to make bold decisions. They chose to revolutionise the journalism industry’s examination system, which had been in place for over 65 years, and successfully delivered a new e-assessment programme including innovative video and editing within 12 months. The selection of a relatively new to market, innovative assessment platform from Cirrus Assessment played a key role in this success.
This submission details how the project was delivered and highlights how a holistic approach, with support for all those involved, has led to great results for learners, examiners, training providers and the NCTJ.
It is a blueprint on how the introduction of technology for high stakes summative assessment can greatly strengthen the authenticity and validity of the exam, whilst also providing tangible benefits to everyone involved. Close collaboration between the NCTJ, Cirrus Assessment, training providers and students coupled with detailed project planning has ensured that all are now toasting its success.
The National Council for the Training of Journalists (NCTJ) is the industry’s charity delivering the premier training scheme for UK journalists. This is provided through a world-class education, training and qualification system developing current and future journalists for the demands of a fast-changing multimedia industry. The NCTJ is regulated by Ofqual, Qualifications Wales and CCEA.
Until 2016, NCTJ’s exams were delivered using traditional pen and paper methods. This presented a number of challenges.
NCTJ qualifications assess the skills required by modern, multimedia journalists. It was increasingly clear that paper based assessments could not effectively mirror modern working practices and the digital environments that modern day journalists operate in.
The logistics associated with paper exams were increasingly complex. Distribution of papers, secure delivery, marking and moderation were also becoming costly.
These difficulties, in turn, presented a risk to the confidentiality and security of exams. There was also potential for longer than desired timelines for the release of results and feedback to unsuccessful students.
In 2015, the NCTJ conducted a comprehensive review of the content and structure of the Diploma in Journalism, their ‘gold standard’ qualification for trainee journalists. This informed the development of the Essential Journalism exam and their decision to modernise their assessment processes. A thorough e-assessment market review was undertaken and the Cirrus Platform was selected for the project.
The Essential Journalism exam is a mandatory assessment in the Diploma in Journalism, designed to assess core journalism skills including reporting using digital platforms, social media, data and analytics.
Planning and Implementation
It was recognised that for the rollout to be successful, multiple stakeholders would need to be engaged.
- internal committees and staff responsible for exam design and development
- external markers and moderators, currently unfamiliar with e-assessment
- training providers and centres apprehensive about using technology for high-stakes exams
A detailed implementation plan was devised to tackle these challenges and activities proposed to assist with transition:
- training programmes for staff including initial training and refresher workshops throughout the year. These would focus on each delivery phase (item creation, assessment creation, pilot delivery, marking and moderation).
- seminars at locations across the country with all centres invited. The NCTJ and Cirrus Assessment would together outline plans, provide updates, demonstrate functionality and answer questions and concerns in open forums.
- pilot exam sessions were scheduled with participating centres across the country. In these, a trial version of the Essential Journalism exam was delivered under exam conditions to students due to take the live exams in November 2016. Members of the NCTJ exams team and Cirrus Assessment staff were present to take feedback from both students and centre administrators. Feedback was given, from both viewpoints on exam format and user experience.
- mock exams were made available, in the Cirrus Platform, to all centres in October 2016 to aid their students’ preparation for the live exam. This also enabled centres to thoroughly test processes and procedures ahead of live delivery.
Live Exam Launch
The Essential Journalism exam is delivered in a secure, lock-down browser-environment that supports the NCTJ in complying with the regulators’ general conditions and its own rigorous exam policies and procedures.
Candidates have the first 30 minutes to read a press release, identify key facts from the story, watch and listen to a simulated video press conference and produce a 140-160 word online news story, headline and social media snippet. This is an innovative and demanding test of their ability to produce work to a deadline and reflects the journalistic skills needed to operate in a multimedia newsroom.
Candidates gave detailed feedback after each exam. This was overwhelmingly positive. They found the user interface extremely intuitive and simple to use and enjoyed using video and audio elements within the assessment.
Additional options are also available to students with visual and auditory impairments. Accessibility can be further enhanced by adding remote invigilation for students with difficulties accessing test centres.
Test centre administrators found the Cirrus Platform very easy to work with and felt well supported by both the NCTJ and Cirrus Assessment. They had access to Cirrus Assessment’s customer success portal where they could find detailed guidance on using the system. Specific guidelines for tutors, trainers, invigilators and students were also provided.
Marking, Moderation, Results and Reporting
On submission, candidate scripts were immediately accessible, in the Cirrus Platform, to the NCTJ exams team. The sophisticated marking workflow assigned scripts to external markers and moderators. Assessors were able to view candidate responses and associated mark schemes within the UI. Markers were able to use onscreen tools to annotate responses which could then be viewed by moderators.
The NCTJ prepared a feedback report using system data for unsuccessful students. This included feedback on questions where they did not score well. This included comments from markers and guidance on where to focus future study. Tutors were able to use this data to inform future planning.
Case study: Chartered Accountants Ireland (CAI)
Tonight, as a huge combined team, we managed to create a truly momentous milestone in the assessment of professional accountancy globally. Chartered Accountants Ireland’s first-ever remote proctored fully regulated exam completed successfully!
Chartered Accountants Ireland (CAI) was established by Royal Charter on 14 May 1888, and is Ireland’s largest accountancy body. It represents more than 25,000 members globally. Qualified members of the Chartered Accountants Ireland earn the designation ACA.
In the autumn of 2018, CAI did a tender process to select a provider for both remote proctoring and end-to-end e-assessment platform. After a thorough process, the choice fell on Cirrus and ProctorU. The partnership stood out from the rest by being innovative and user-friendly and give CAI the confidence that they would be well taken care of.
Chartered Accountants Ireland is a prestigious organisation dating back to before the Irish republic and were traditional in the way they approached assessments. But the costs of test centres forced them to look at other venues for exam delivery. Together with their sister organisation they deliver in excess of 35,000 exams a year.
Cirrus signed the contract with the Chartered Accountants in January 2019, with the deadline of delivering their first exams end of March. And that with the need to develop 2 bespoke question types for them. Cirrus has never backed down from a challenge and started immediately with gathering the team for CAI. The training has been done for CAI and our consultants are working with them to implement the platform. At the same time, a team of developers are working on the new question types. At Cirrus we understand that it’s much at stake for CAI, both reputational and economic. We have therefore given a guarantee that the implementation and bespoke development will be ready in time.